{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ab8aa46c-91b1-5413-9ab8-8452ddd88cb3","task_key":"all--test--ab8aa46c-91b1-5413-9ab8-8452ddd88cb3","task_revision_id":"1","upstream_id":"","short_description":"A man was arrested and convicted of publishing paid newspaper advertisements…","config":"all","split":"test","body":"{\"choices\":\"['No, the statute violates the sixth amendment right to privacy.', 'No, the statute violates the First Amendment right to publish protected commercial speech.', 'Yes, the right to an abortion does not include the right to publicize where to get one.', 'Yes, there is no right to publish or disseminate speech that is commercial in nature.']\",\"question\":\"A man was arrested and convicted of publishing paid newspaper advertisements giving information on how to get an abortion in another state and how to find a referral service. A criminal statute made it a crime to disseminate such information. Will the courts uphold the conviction?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}