{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c35f5134-1ba9-591a-9e82-8e352d0dd05c","task_key":"all--test--c35f5134-1ba9-591a-9e82-8e352d0dd05c","task_revision_id":"1","upstream_id":"","short_description":"Recent discoveries of Homo naledi fossils in the Dinaledi Chamber of the Rising…","config":"all","split":"test","body":"{\"choices\":\"['there are already two well-known species of Homo and it is unlikely there were more than two species at one time.', 'paleoanthropologists did not make the discovery and it was found in an inaccessible cave in South Africa.', 'there is, as yet, no firm date that can be applied to the specimens.', 'the Homo naledi fossils have been shown to pre-date the earliest known hominid fossils by millions of years.']\",\"question\":\"Recent discoveries of Homo naledi fossils in the Dinaledi Chamber of the Rising Star cave system in South Africa suggest a mixture of post-cranial morphologies or mosaic of hominid traits. One of the reasons this new species is controversial is that:\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}