{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f4910073-f10a-5c28-8b8f-797d39fc9dca","task_key":"all--test--f4910073-f10a-5c28-8b8f-797d39fc9dca","task_revision_id":"1","upstream_id":"","short_description":"One evening, a husband received a telephone call from an unidentified source who…","config":"all","split":"test","body":"{\"choices\":\"[\\\"guilty of attempted murder of his friend's wife only.\\\", 'guilty of attempted murder of his friend only.', \\\"guilty of attempted murder of both his friend and his friend's wife.\\\", \\\"not guilty of attempted murder of either his friend or his friend's wife.\\\"]\",\"question\":\"One evening, a husband received a telephone call from an unidentified source who informed him that his wife was having an affair with his friend. As a result, the husband drove to his friend's house with a loaded shotgun. Believing that his friend was in an upstairs bedroom, the husband fired the shotgun through the dining room window, as he only wanted to teach him a lesson. Unbeknownst to the husband, his friend's wife was in the dining room and suffered a minor gunshot wound. The husband should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}