{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1f8a106a-678d-59f4-abb2-6079f2ddb964","task_key":"all--test--1f8a106a-678d-59f4-abb2-6079f2ddb964","task_revision_id":"1","upstream_id":"","short_description":"A state criminal law prohibits the publication of any description of the details…","config":"all","split":"test","body":"{\"choices\":\"['No, because the prosecution seeks to punish the publication of lawfully obtained, truthful information about a matter of public significance, without adequate justification.', \\\"No, because the reporter did not promise prison officials that he would report only the warden's official statement about the execution.\\\", 'Yes, because publication of the details of such events might cause psychological damage to some children.', \\\"Yes, because the newspaper should have brought an action to test the validity of the law before publishing the reporter's story.\\\"]\",\"question\":\"A state criminal law prohibits the publication of any description of the details of the execution of any prisoner who is sentenced to death by the courts of the state. Although the law allows a specified number of reporters to observe an execution, only the prison warden's official statement that the prisoner was \\\"executed as provided by law\\\" at a certain time and date may be published. The purpose of the law is to protect the public, particularly children, from the details of executions. After a particular execution was mishandled, causing the prisoner to suffer, a newspaper in the state published a story describing the event in detail. The story was written by the newspaper's reporter, who was permitted to observe the execution but did not promise prison officials that he would report only the warden's official statement. A prosecutor subsequently filed charges against the newspaper for publishing the details of the execution in violation of the state law. Is this prosecution constitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}