{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ad89d1bd-bd3d-552a-9176-75743ca0d158","task_key":"all--test--ad89d1bd-bd3d-552a-9176-75743ca0d158","task_revision_id":"1","upstream_id":"","short_description":"A new gang member, hoping to impress the gang's leader, pointed a gun at a…","config":"all","split":"test","body":"{\"choices\":\"['Accessory after the fact to robbery.', 'Accomplice to robbery.', 'Conspiracy to commit robbery.', 'Receiving stolen property.']\",\"question\":\"A new gang member, hoping to impress the gang's leader, pointed a gun at a pedestrian and ordered her to give him her expensive watch, which she did. The gang member then tossed the watch to the gang leader, who was standing nearby. Although totally surprised by this act, the gang leader put the watch in his pocket. The pedestrian ran away. What crime did the gang leader commit?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}