{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"de54222a-09af-5cdd-badf-0b39421c25ca","task_key":"all--test--de54222a-09af-5cdd-badf-0b39421c25ca","task_revision_id":"1","upstream_id":"","short_description":"A defendant had been drinking at a bar for three hours and was visibly…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because his intoxication was voluntary.', 'guilty, because he was under a duty to retreat.', 'not guilty, because of his intoxication.', 'not guilty, because there is no duty to retreat in a public place.']\",\"question\":\"A defendant had been drinking at a bar for three hours and was visibly intoxicated. A man entered the bar and sat down next to the defendant. After ordering a beer, the man turned to the defendant and said, \\\"Hey buddy, you're sure an ugly looking dude. \\\" The defendant ignored the man's insult and turned to walk away. The man then pushed the defendant against the bar and said, \\\"Your face makes me sick to my stomach. \\\" The defendant then pulled out a razor and slashed the man's throat, killing him. If the defendant is prosecuted for the man's murder, he will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}