{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"74e6ef98-6c9f-50a5-8e5a-d8e1cf123a87","task_key":"all--test--74e6ef98-6c9f-50a5-8e5a-d8e1cf123a87","task_revision_id":"1","upstream_id":"","short_description":"New York State has a criminal statute making it illegal to knowingly interfere…","config":"all","split":"test","body":"{\"choices\":\"['Reject the instruction, based on ex post facto.', 'Reject the instruction as a violation of due process.', 'Admit the instruction, because the law was passed before the prosecution was initiated.', 'Admit the instruction, as a bill of attainder.']\",\"question\":\"New York State has a criminal statute making it illegal to knowingly interfere with the operation of government with malicious intent. Corry is a computer hacker, who shut down the New York government's website by infecting it with a virus. The virus entered the electronic mail of users and mailed itself to every address listed in each user's e-mail address book every two hours. The website was crippled for the last week of January. The state passed a law in February clarifying that knowingly infecting the state's internet server with a computer virus met the definition of interference in the statute. During the subsequent criminal trial against Corry in March, the state asked for a jury instruction based on the statutory computer virus clarification. The defense objects to the instruction. How should the state court judge rule?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}