{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c39e30f0-47ae-5b7f-9c0a-5cfaf5b2c521","task_key":"default--test--10356","task_revision_id":"1","upstream_id":"10356","short_description":"A method is to be written to search an array for a value that is larger than a…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['The method should return an error if more than one larger value is found.', 'The specification should be modified to indicate what should be done if there is more than one index of larger values.', 'The method should be written to output a message if more than one larger value is found.', 'The method should be written so as to return the index of every occurrence of a larger value.', 'The method should be written to return the last occurrence of a larger value.', 'The method should return a random index of a larger value.', 'The method should be written to return the largest index among larger values.', 'The method should be written on the assumption that there is only one value in the array that is larger than the given item.']\",\"question\":\"A method is to be written to search an array for a value that is larger than a given item and return its index. The problem specification does not indicate what should be returned if there are several such values in the array. Which of the following actions would be best?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}