{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6b20cffd-2e15-5a7d-98b2-29edcaf4b00c","task_key":"all--test--6b20cffd-2e15-5a7d-98b2-29edcaf4b00c","task_revision_id":"1","upstream_id":"","short_description":"A man and a brother were identical twins. The man was having an affair with a…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the brother did not inform him of his mistake.', \\\"prevail, because the husband's threat was accompanied by an overt act when the husband approached the brother.\\\", 'not prevail, because the husband was the initial aggressor.', 'not prevail, because the brother used reasonable force given the threat.']\",\"question\":\"A man and a brother were identical twins. The man was having an affair with a woman. The woman's husband was aware of his wife's relationship with the man. One morning the husband saw the brother, whom he mistakenly believed was the man. The husband walked toward the brother and said, \\\"I'm going to punch you in the face, you rat. \\\" The brother immediately realized that the husband had mistaken him for his brother. The brother had enough time to inform the husband of his mistake but didn't say anything. When the husband came closer, the brother punched him in the mouth. If the husband brings suit against the brother for battery, he should\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}