{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed0b680b-2902-599d-9932-63e7b5e6037a","task_key":"all--test--ed0b680b-2902-599d-9932-63e7b5e6037a","task_revision_id":"1","upstream_id":"","short_description":"A defendant and a victim were roommates at college. The defendant was playing…","config":"all","split":"test","body":"{\"choices\":\"['guilty of murder.', 'guilty of manslaughter.', 'guilty of battery.', 'not guilty.']\",\"question\":\"A defendant and a victim were roommates at college. The defendant was playing the role of a serial killer in the school play and asked the victim if she could leave their apartment for a few hours while she rehearsed. The victim agreed but returned to the apartment shortly thereafter because she was curious and wanted to see the defendant rehearse. The victim quietly came back into the apartment and hid behind some curtains in the dining room in order to watch the defendant rehearse. In her role, the defendant earned a large butcher's knife, with which she stabbed the curtains at her \\\"victim. \\\" Unknown to the defendant, the victim, who was hiding behind the curtain, was stabbed in the chest and killed. If the defendant is prosecuted for the victim's death, she should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}