{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c0217259-d8d5-5d39-a891-514f1c072ead","task_key":"default--test--5007","task_revision_id":"1","upstream_id":"5007","short_description":"The transition from food gathering to a total reliance on agriculture:","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['was driven solely by population growth.', 'only occurred in Europe and Asia.', 'took hundreds of thousands of years.', 'always required careful planning.', 'happened everywhere except in South America.', 'was not possible without domesticated animals.', 'was a direct result of advancements in technology.', 'happened overnight.', 'only took place in societies with a written language.', 'took thousands of years.']\",\"question\":\"The transition from food gathering to a total reliance on agriculture:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}