{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e53d59dd-d224-5bb9-abda-34859d9ed082","task_key":"default--test--4964","task_revision_id":"1","upstream_id":"4964","short_description":"Which of the following characteristics of modern humans is the most ancient,…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['establishment of permanent settlements', 'development of agriculture', 'domestication of animals', 'creation of written language', 'symbolic expression through the production of art', 'development of complex social hierarchies', 'elaborate burials', 'improved stone tool technologies', 'larger sites with larger populations']\",\"question\":\"Which of the following characteristics of modern humans is the most ancient, pre-dating the Late Stone Age, the Upper Paleolithic and even anatomically modern humans?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}