{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5021588e-0daa-5234-95d3-7cd482123b47","task_key":"all--test--5021588e-0daa-5234-95d3-7cd482123b47","task_revision_id":"1","upstream_id":"","short_description":"A defendant was wearing a black leather jacket when he entered a deli for lunch.…","config":"all","split":"test","body":"{\"choices\":\"['there was a mistake of fact.', 'he returned the jacket after discovering his mistake.', 'he lacked the requisite state of mind.', 'there was no fraudulent conversion.']\",\"question\":\"A defendant was wearing a black leather jacket when he entered a deli for lunch. He placed his jacket on a coat rack located in the front of the deli. After his meal, the defendant picked up a similar black leather jacket, believing it to be his own. The jacket he took, however, belonged to another customer. The defendant left the deli and walked a short distance before realizing he had taken the wrong jacket by mistake. He then returned to the deli and placed the jacket back on the coat rack. He found his own jacket, which had been partially hidden under a stack of other coats. If the defendant were arrested and charged with larceny of the other customer's jacket, he will most likely be acquitted because\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}