{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f209241c-4304-50e4-bc96-447df57632fc","task_key":"all--test--f209241c-4304-50e4-bc96-447df57632fc","task_revision_id":"1","upstream_id":"","short_description":"A defendant was driving his car recklessly at a high rate of speed through a…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because a person is presumed to intend the natural and probable consequences of his acts.', \\\"guilty, because criminal liability is predicated upon the defendant's willful and wanton disregard for the safety of others.\\\", 'not guilty, because the defendant did not intend to kill the woman.', 'not guilty, because he lost control of the vehicle.']\",\"question\":\"A defendant was driving his car recklessly at a high rate of speed through a residential neighborhood. He was traveling at a speed of over 100 M. P. H. when he lost control of the car and jumped a curb, striking a woman who was walking along the sidewalk. As a result of the collision, the woman suffered severe internal injuries and fractured both legs. She was hospitalized for 11 months and became permanently disabled. If the defendant is charged with attempted murder, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}