{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f0c313f-825f-5ef6-a4ca-c8ed6f6fa1e9","task_key":"default--test--4686","task_revision_id":"1","upstream_id":"4686","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['United States v. Nixon (1974)', 'Miranda v. Arizona (1966)', 'Gideon v. Wainwright (1963)', 'Marbury v. Madison (1803)', 'Brown v. Board of Education of Topeka (1954)', 'Ex parte Milligan (1866)', 'Roe v. Wade (1973)', 'Plessy v. Ferguson (1896)', 'McCulloch v. Maryland (1819)', 'Dred Scott v. Sanford (1857)']\",\"question\":\"This question refers to the following information.\\n\\\"The question, therefore, should be quickly settled, whether free colored persons, born and naturalized in this country, are not American citizens, and justly entitled to all the rights, privileges and immunities of citizens of the several states; and whether the Constitution of the United States makes or authorizes any invidious distinction with regard to the color or condition of free inhabitants.\\n\\\"For myself, I have not the shadow of doubt on the subject. I believe that the rights of the free colored persons need only to be vindicated before the U.S. Supreme Court, to be obtained; that no prejudice or sophistry . . . can prevent their acknowledgement . . . and that the present laws, affecting your condition, are clearly unconstitutional. The fact that you have been treated, by common consent and common usage, as aliens and brutes, is not proof that such treatment is legal, but only shows the strength, the bitterness, and the blindness of prejudice.\\\"\\n—William Lloyd Garrison, \\\"To the Free People of Color of the United States,\\\" The Liberator, January 15, 1831\\nThe argument put forth by William Lloyd Garrison in the passage was later contradicted in which of the following Supreme Court decisions?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}