{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1e1ba59-a963-5239-b8bf-13c6826a6b7c","task_key":"all--test--d1e1ba59-a963-5239-b8bf-13c6826a6b7c","task_revision_id":"1","upstream_id":"","short_description":"A woman had been spreading rumors around town that a defendant was a shoplifter.…","config":"all","split":"test","body":"{\"choices\":\"[\\\"guilty, because she broke into the woman's home at night with the intent to commit a felony therein.\\\", 'guilty, because the woman suffered a broken jaw as a result of the blow inflicted by the defendant.', 'not guilty, because the defendant did not intend to seriously injure the woman.', 'not guilty, because the defendant did not intend to commit a felony at the time of the breaking and entry.']\",\"question\":\"A woman had been spreading rumors around town that a defendant was a shoplifter. After the defendant learned about these rumors, she was furious and decided that she would teach the woman to keep her mouth shut. Late that night, the defendant went over to the woman's house with the intent to punch her in the mouth. When the defendant arrived at the woman's home, she peered in the bedroom window and saw the woman asleep in bed. The defendant proceeded to pry open the bedroom window and climbed inside. The defendant then grabbed the woman and punched her in the face and hurriedly fled from the home. The force of the blow fractured the woman's jaw. If the defendant is subsequently prosecuted for burglary, she will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}