{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6eacb8f8-4a61-58fd-b0c1-13f274c9a685","task_key":"all--test--6eacb8f8-4a61-58fd-b0c1-13f274c9a685","task_revision_id":"1","upstream_id":"","short_description":"A defendant hated a victim and decided to kill him. The defendant put a loaded…","config":"all","split":"test","body":"{\"choices\":\"['first-degree murder.', 'second-degree murder.', 'involuntary manslaughter.', 'voluntary manslaughter.']\",\"question\":\"A defendant hated a victim and decided to kill him. The defendant put a loaded gun in his coat pocket and went outside for a walk, not sure if and when he may see the victim. The defendant passed by a bar and decided to go inside for a drink. While in the bar, he drank several bottles of beer and became intoxicated. While walking home from the bar, the defendant bumped into the victim. As soon as the defendant recognized him, he pulled out his gun and shot and killed the victim. In this jurisdiction, first-degree murder is defined as a premeditated and deliberate killing. All other forms of murder are second-degree murder. The defendant is guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}