{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1d7ce640-0179-54f4-87ba-2cce440bbde1","task_key":"all--test--1d7ce640-0179-54f4-87ba-2cce440bbde1","task_revision_id":"1","upstream_id":"","short_description":"A husband came home early from work one day. He walked to the bedroom and, upon…","config":"all","split":"test","body":"{\"choices\":\"['Murder.', 'Voluntary manslaughter.', 'Involuntary manslaughter.', 'No crime.']\",\"question\":\"A husband came home early from work one day. He walked to the bedroom and, upon opening the door, saw his wife and another man in bed together. Visibly upset, the husband told the other man to leave. As the man was reaching for his pants, he pulled out a gun, fearing that the husband was about to attack him. When the husband saw the gun, he jumped on the man and angrily twisted his neck, breaking it and killing him. The husband is guilty for which, if any, of the following crimes?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}