{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"60d03bf8-fa25-5d0d-ab39-c3754f88ae0f","task_key":"all--test--60d03bf8-fa25-5d0d-ab39-c3754f88ae0f","task_revision_id":"1","upstream_id":"","short_description":"The commoners of the rural Aztec villages of Capilco and Cuexcomate:","config":"all","split":"test","body":"{\"choices\":\"['were ruled by an iron fist by the king, who had armies stationed there.', 'were forced to send half of all maize they grew to the king as tribute and lived in absolute poverty.', 'were mostly left alone and did fairly well for themselves.', 'had no protection from city states that invaded from the north.']\",\"question\":\"The commoners of the rural Aztec villages of Capilco and Cuexcomate:\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}