{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bc516b2f-ecf7-57d6-85ff-5008b181df7b","task_key":"all--test--bc516b2f-ecf7-57d6-85ff-5008b181df7b","task_revision_id":"1","upstream_id":"","short_description":"Up to 50,000 years ago, what was the relationship between anatomically modern…","config":"all","split":"test","body":"{\"choices\":\"['They looked and thought like us.', 'They looked like us but did not think like us.', 'They thought like us but did not look like us.', 'They did not look or think like us.']\",\"question\":\"Up to 50,000 years ago, what was the relationship between anatomically modern humans and us?\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}