{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"29a9cc8c-895f-5400-844f-96478fd5d1a6","task_key":"default--test--8446","task_revision_id":"1","upstream_id":"8446","short_description":"A talk show host recently reported that in response to his on-air question, 82%…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['The survey is biased because it only includes listeners who have access to email.', 'The survey would have been more meaningful if he had used a control group.', 'The survey is accurate as it includes a large number of responses.', 'The survey would have been more meaningful if he had picked a random sample of the 2500 listeners who responded.', \\\"The survey is valid because it represents the opinion of the show's listeners.\\\", 'The survey would have been more meaningful if he had included listeners who disagreed with his viewpoint.', 'The survey is meaningless because of voluntary response bias.', \\\"The survey shows that the majority of the host's listeners support the death penalty for drug dealers.\\\"]\",\"question\":\"A talk show host recently reported that in response to his on-air question, 82% of the more than 2500 e-mail messages received through his publicized address supported the death penalty for anyone convicted of selling drugs to children. What does this show?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}