{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7676d044-9f7f-58af-aace-cac9e39f6863","task_key":"all--test--7676d044-9f7f-58af-aace-cac9e39f6863","task_revision_id":"1","upstream_id":"","short_description":"Howard did not know it, but his wife was having an affair with another man. One…","config":"all","split":"test","body":"{\"choices\":\"['guilty of manslaughter of his wife, but not guilty of manslaughter of the other man.', 'guilty of manslaughter of the other man, but not guilty of manslaughter of his wife.', 'guilty of manslaughter of both his wife and the other man.', 'not guilty of manslaughter of either his wife or the other man.']\",\"question\":\"Howard did not know it, but his wife was having an affair with another man. One day Howard came home early from work and caught his wife and the other man in bed together. In a violent rage, Howard picked up a large metal lamp and severely beat his wife and the other man. Both victims were seriously injured and hospitalized. Howard was charged with the attempted murder of the other man. He was acquitted of the crime. After Howard's acquittal, his wife and the other man died from injuries that were related to the beating. The state now wishes to prosecute Howard for manslaughter for the deaths of his wife and the other man. Howard should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}