{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3de574ab-44f7-5db6-9883-672ad5fd7c14","task_key":"all--test--3de574ab-44f7-5db6-9883-672ad5fd7c14","task_revision_id":"1","upstream_id":"","short_description":"Two college students were members of the same fraternity. During a fraternity…","config":"all","split":"test","body":"{\"choices\":\"['guilty of assault, but not murder or manslaughter.', 'guilty of manslaughter.', 'guilty of second-degree murder.', 'guilty of first-degree murder.']\",\"question\":\"Two college students were members of the same fraternity. During a fraternity party, the defendant pulled out a gun, pointed it at the other student and said, \\\"Beg for your life, sucker. \\\" The student, who knew that the defendant had a reputation as a practical joker, said, \\\"Get lost, you jerk. \\\" The defendant then pulled the trigger and shot the student to death. A statute in this jurisdiction provides: \\\"Any intentional and premeditated killing with malice aforethought is murder in the first degree. Murder in the second degree is any killing that occurs during the commission or attempted commission of a serious or inherently dangerous felony. Manslaughter includes all other types of unlawftil homicide and unjustifiable killings. \\\"The defendant was subsequently prosecuted for killing the student. At trial, the defendant testified that a fellow fraternity member gave him the gun before the shooting and told him that it was unloaded. If the jury believes the defendant, it should find him\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}