{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"50b80229-cbff-55ed-afbf-92396a10695c","task_key":"all--test--50b80229-cbff-55ed-afbf-92396a10695c","task_revision_id":"1","upstream_id":"","short_description":"When anatomically modern humans first arrived in the Middle East, who did they…","config":"all","split":"test","body":"{\"choices\":\"['Neandertals, the evolutionary descendants of the premodern human inhabitants of Europe and Asia', 'Homo erectus, the evolutionary descendants of the premodern human inhabitants of Europe and Asia', 'Homo sapiens, who had evolved independently in Asia and arrived there by 200,000 years ago', 'All of the above']\",\"question\":\"When anatomically modern humans first arrived in the Middle East, who did they encounter?\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}