{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58684526-4c71-51b0-b5b0-4015045e05a6","task_key":"all--test--58684526-4c71-51b0-b5b0-4015045e05a6","task_revision_id":"1","upstream_id":"","short_description":"A company created a drug to lower high blood pressure. The President issues an…","config":"all","split":"test","body":"{\"choices\":\"['valid, because the President has the authority to ensure that laws are faithfully executed.', 'valid, because the President has the authority to impose economic regulations unless overruled by Congress.', 'invalid, because it is an unauthorized extension of executive power.', 'invalid, because the President does not have the power to regulate interstate commerce.']\",\"question\":\"A company created a drug to lower high blood pressure. The President issues an executive order prohibiting the shipment and sale of the drug within the United States. Under this executive order, federal agents are empowered to prosecute all interstate shippers and sellers of the drug. The President claims that he is acting upon conclusive evidence that the drug causes cervical cancer in laboratory monkeys. This executive order is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}