{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"92b25ea9-1d66-56ef-b45d-e83556d2ef25","task_key":"all--test--92b25ea9-1d66-56ef-b45d-e83556d2ef25","task_revision_id":"1","upstream_id":"","short_description":"The defendant was an off-duty police officer who frequently worked security at…","config":"all","split":"test","body":"{\"choices\":\"['guilty.', 'not guilty, because the killing constituted justifiable homicide.', 'not guilty, because the killing constituted excusable homicide.', 'not guilty, because the defendant warned the groupie before he fired his gun.']\",\"question\":\"The defendant was an off-duty police officer who frequently worked security at concerts on his free time. One night, he was working backstage security for one of his favorite bands when he noticed a groupie sneak in the back door, pick up a guitar, and begin to walk out with it. He drew his service revolver and exclaimed, \\\"Stop or I'll shoot!\\\" The groupie looked at him and then fled. The defendant fired at the fleeing groupie, fatally wounding her. Use of force was not authorized by the concert promotion company that employed him. On these facts, if the defendant is charged with murder, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}