{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"68249a49-910d-5637-8faa-79fcb7e0d3d1","task_key":"all--test--68249a49-910d-5637-8faa-79fcb7e0d3d1","task_revision_id":"1","upstream_id":"","short_description":"Experiments with stone spear points made to replicate those artifacts found at…","config":"all","split":"test","body":"{\"choices\":\"['made and used spears with stone projectile points for hunting 500,000 years ago.', 'made and used spears with stone projectile points for hunting more than 3 million years ago.', 'did not use spears with stone projectile points, but probably hunted and caught fish with wooden and bone tools.', 'made stone spear points for use as weapons, but not for hunting.']\",\"question\":\"Experiments with stone spear points made to replicate those artifacts found at the Kathu Pan site in South Africa suggest that Homo erectus:\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}