{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"763fc02f-aa8e-52c4-8b84-429ef9f04fa9","task_key":"default--test--8234","task_revision_id":"1","upstream_id":"8234","short_description":"On the same test, Mary and Pam scored at the 64th and 56th percentiles,…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['36% of the test takers scored lower than Pam.', 'Eight percent of those who took the test scored between Pam and Mary.', 'Mary scored 64% of the total points and Pam scored 56% of the total points.', 'Mary scored eight more points than Pam.', \\\"Mary's score is 8% higher than Pam's.\\\", 'Mary scored 64 points and Pam scored 56 points.', '56 people scored lower than Mary.', 'Mary and Pam are in the top 64th and 56th percent of all students respectively.', 'Thirty-six people scored higher than both Mary and Pam.', 'Eight people scored higher than Mary and Pam.']\",\"question\":\"On the same test, Mary and Pam scored at the 64th and 56th percentiles, respectively. Which of the following is a true statement?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}