{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"48d90346-21e5-55dd-8bd5-8a1b7d764cb7","task_key":"default--test--8346","task_revision_id":"1","upstream_id":"8346","short_description":"Two classes take the same exam. Suppose a certain score is at the 40th…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Students in the first class generally scored higher than students in the second class.', 'A score at the 50th percentile for the first class is at the 90th percentile for the second class.', 'The average score of the first class is the same as the average score of the second class.', 'A score at the 60th percentile for the first class is at the 80th percentile for the second class.', 'Students in the second class generally scored higher than students in the first class.', 'A score at the 40th percentile for the first class is at the 20th percentile for the second class.', 'A score at the 20th percentile for the first class is at the 40th percentile for the second class.', 'A score at the 80th percentile for the first class is at the 40th percentile for the second class.', 'There is no correlation between the scores of the two classes.', 'Students in the first class scored the same as students in the second class.']\",\"question\":\"Two classes take the same exam. Suppose a certain score is at the 40th percentile for the first class and at the 80th percentile for the second class. Which of the following is the most reasonable conclusion?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}