{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"22b4e976-e57a-5621-b07d-9d3466f6ee9f","task_key":"all--test--22b4e976-e57a-5621-b07d-9d3466f6ee9f","task_revision_id":"1","upstream_id":"","short_description":"The victim owned and operated a small convenience store that was situated on the…","config":"all","split":"test","body":"{\"choices\":\"['guilty of felony murder.', 'guilty of involuntary manslaughter.', 'guilty of voluntary manslaughter.', 'not guilty, because it was unforeseeable that a death would occur under the circumstances.']\",\"question\":\"The victim owned and operated a small convenience store that was situated on the first floor of the building in which the victim occupied a second-floor apartment. Late one evening, the defendant was walking past the convenience store when he decided to break into the store to steal beer and cigarettes. The defendant threw a brick through the window of the convenience store and quickly went inside. Awakened by the alarm, the victim went down the stairs and into the convenience store. Seeing the victim, the defendant turned and began to run. Seeing the defendant running away with a case of beer and a carton of cigarettes, the victim ran after the defendant but tripped and fell, breaking his neck, which resulted in his immediate death. If the defendant is subsequently charged with the victim's death, he will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}