{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4ff818a-ea8e-5f4a-8778-120685dd6b9e","task_key":"default--test--4824","task_revision_id":"1","upstream_id":"4824","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Twenty-first Amendment, which repealed the prohibition of alcohol', 'The Fourteenth Amendment, which granted citizenship to all persons born or naturalized in the United States', 'The Nineteenth Amendment, which guaranteed all women the right to vote', 'The Fifth Amendment, which protects against self-incrimination', 'The Second Amendment, which guaranteed the right to bear arms', 'The Twenty-sixth Amendment, which lowered the voting age to 18', 'The First Amendment, which guaranteed the right to free speech', 'The Thirteenth Amendment, which abolished slavery', 'The Seventeenth Amendment, which established the direct election of senators', 'The Tenth Amendment, which allows powers not granted to the federal government be granted to the states']\",\"question\":\"This question refers to the following information.\\n\\\"Some men look at constitutions with sanctimonious reverence, and deem them like the ark of the covenant, too sacred to be touched. They ascribe to the men of the preceding age a wisdom more than human, and suppose what they did to be beyond amendment….But I know also, that laws and institutions must go hand in hand with the progress of the human mind. As that becomes more developed, more enlightened, as new discoveries are made, new truths disclosed, and manners and opinions change with the change of circumstances, institutions must advance also, and keep pace with the times.\\\"\\nThomas Jefferson, 1816\\nWhich of following Amendments to the Constitution is most directly an example of the sentiments expressed above?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}