{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fc686ed5-d79c-5c02-8ff1-77464279e607","task_key":"all--test--fc686ed5-d79c-5c02-8ff1-77464279e607","task_revision_id":"1","upstream_id":"","short_description":"A man is on trial for rape. The alleged victim testified that she went out to…","config":"all","split":"test","body":"{\"choices\":\"['not guilty, because he honestly believed that the victim consented.', 'not guilty, because his intoxication negated his criminal intent.', 'guilty, because rape is a general intent crime.', 'guilty, because she did not consent, and his belief that she was consenting was unreasonable.']\",\"question\":\"A man is on trial for rape. The alleged victim testified that she went out to dinner with the man. Afterward, he invited her to his apartment for coffee. Upon entering the apartment, he violently assaulted her. Although she tried to resist, he overpowered and raped her. The man testified that during dinner, he and the victim drank two bottles of Champagne. When they returned to his apartment, he was so intoxicated that he honestly believed that she consented to the intercourse. The jury determined that the victim did not consent to the intercourse. The jury also found that the man, as a result of his intoxication, honestly but unreasonably believed that she was consenting. As a consequence, the defendant should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}