{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d6a8785-3f4a-5544-8549-a27eeb233fc8","task_key":"all--test--9d6a8785-3f4a-5544-8549-a27eeb233fc8","task_revision_id":"1","upstream_id":"","short_description":"A valid treaty between the United States and a foreign country provides for the…","config":"all","split":"test","body":"{\"choices\":\"['No, because the Constitution vests in Congress the exclusive authority to specify binding legal standards for weights and measures, and the President therefore lacks constitutional authority for the proclamation.', 'No, because the principles of federalism embedded in the Constitution prohibit the President from taking action to invalidate a state law.', 'Yes, because it is authorized by a valid treaty of the United States and is not prohibited by any provision of the Constitution and, therefore, is the supreme law of the land.', 'Yes, because the President has inherent authority to nullify any state law that substantially impedes commerce between substantially impedes commerce between the United States and another country.']\",\"question\":\"A valid treaty between the United States and a foreign country provides for the elimination of all tariff barriers between the two countries. It authorizes the president of either country to issue a proclamation nullifying any state or local laws in that country that have the effect of impeding imports from the other country. The foreign country uses the metric system of measurement, and thus all goods produced there and exported to the United States are packaged in metric sizes, such as liters and kilograms. A law of a state in the United States requires all goods sold in that state to be packaged in traditional American sizes, such as quarts or pounds. Because the state law substantially impedes imports from the foreign country, the President of the United States has issued a proclamation nullifying the state law pursuant to the treaty. Is the President's proclamation valid?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}