{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2ce775dd-6180-52c3-a18a-64286c89a4f4","task_key":"all--test--2ce775dd-6180-52c3-a18a-64286c89a4f4","task_revision_id":"1","upstream_id":"","short_description":"Based on the morphology of fossil hominids dating from about 1.8 million until…","config":"all","split":"test","body":"{\"choices\":\"['there were two or more related but geographically separate hominid species during this period that evolved from earlier African hominids', 'hominids dating from this time can be grouped into different species in Africa, Asia, and Europe.', 'all of the hominids that evolved from Homo habilis and predate Homo sapiens belong in a single, geographically widespread but morphologically diverse species known as Homo erectus.', 'all of the above']\",\"question\":\"Based on the morphology of fossil hominids dating from about 1.8 million until about 400,000 years ago, researchers argue that:\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}