{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"078c9a2f-975e-5cf6-8060-961356318b98","task_key":"all--test--078c9a2f-975e-5cf6-8060-961356318b98","task_revision_id":"1","upstream_id":"","short_description":"The defendant, who was walking on a public street, got into a heated argument…","config":"all","split":"test","body":"{\"choices\":\"['Yes, his use of a deadly weapon demonstrated the requisite intent to kill.', \\\"No, defendant responded to the victim's provocation and did not intend to kill.\\\", 'No, he acted on the spur of the moment and could not have formed the requisite intent.', 'No, his behavior was reckless but not intentional.']\",\"question\":\"The defendant, who was walking on a public street, got into a heated argument with a stranger who had been singing and shouting obscenities while walking near him. The defendant pulled a large serrated hunting knife and stabbed the victim in the chest, killing him. The authorities charged defendant with murder. The defendant argued at trial that he only intended to scare the victim to shut him up, and that he did not have the intent to kill. Is the defendant guilty of murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}