{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e3a66d2-76f2-56a4-bd22-873eb42a06e1","task_key":"default--test--8465","task_revision_id":"1","upstream_id":"8465","short_description":"A shopping mall wants to conduct a survey of the people who shop at the mall.…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['conducting the survey at one shoe store', \\\"conducting the survey online targeting only the mall's registered customers\\\", 'conducting the survey at all clothing stores', 'conducting the survey only on weekends', 'conducting the survey at the entrance to the mall', 'conducting the survey in the mall parking lot', 'conducting the survey at all shoe stores', 'conducting the survey a mile away from the mall ', 'conducting the survey at one specific time of the day', \\\"conducting the survey at the mall's food court\\\"]\",\"question\":\"A shopping mall wants to conduct a survey of the people who shop at the mall. Which would give them the most representative sample?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}