{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"78648b63-1d2c-5596-b05d-7f080de68367","task_key":"default--test--4738","task_revision_id":"1","upstream_id":"4738","short_description":"After 400,000 years ago, hominid evolution is characterized by:","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the development of language and writing.', 'slowly increased brain size and the first stone tools.', 'a rapid increase in brain size.', 'the transition from bipedalism to quadrupedalism.', 'a decrease in stature and increase in brain size.', 'bipedalism and relatively stable brain size.', 'a slight decrease in average brain size.', 'a rapid increase in body size.', 'a rapid decrease in brain size.', 'the development of complex social structures and no change in brain size.']\",\"question\":\"After 400,000 years ago, hominid evolution is characterized by:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}