{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"967b18ac-4d97-544f-9f53-05def17eff5e","task_key":"all--test--967b18ac-4d97-544f-9f53-05def17eff5e","task_revision_id":"1","upstream_id":"","short_description":"In a tavern, an intoxicated woman threatened to slash a man with a broken beer…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, because the customer's conduct was privileged as a defense of others.\\\", 'Yes, based on battery only.', 'Yes, based on false imprisonment only.', 'Yes, based on both battery and false imprisonment.']\",\"question\":\"In a tavern, an intoxicated woman threatened to slash a man with a broken beer bottle. Another customer, who had not been threatened by the woman, forcefully grabbed the woman and locked her in the tavern's storeroom until the police could arrive. In the process, although the customer used reasonable force, the customer badly sprained the woman's wrist. Is the woman likely to recover in an action against the customer?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}