{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3d7f1b2-5cfa-50f3-a2a2-70cf5fd3dfab","task_key":"default--test--5017","task_revision_id":"1","upstream_id":"5017","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['an expression of Transcendentalist individualism.', 'a representation of Enlightenment thinking.', 'a continuation of the ideals of the First Great Awakening.', 'a reflection of literary romanticism.', 'a rejection of the idea of \\\"Republican motherhood.\\\"', 'an outgrowth of the Second Great Awakening.', 'a manifestation of the principles of the Abolitionist movement.', 'an echo of the sentiments of the American Revolution.', \\\"an example of the influence of the Women's Suffrage movement.\\\", 'an embodiment of the ideals of the Temperance movement.']\",\"question\":\"This question refers to the following information.\\n\\\"The question, therefore, should be quickly settled, whether free colored persons, born and naturalized in this country, are not American citizens, and justly entitled to all the rights, privileges and immunities of citizens of the several states; and whether the Constitution of the United States makes or authorizes any invidious distinction with regard to the color or condition of free inhabitants.\\n\\\"For myself, I have not the shadow of doubt on the subject. I believe that the rights of the free colored persons need only to be vindicated before the U.S. Supreme Court, to be obtained; that no prejudice or sophistry . . . can prevent their acknowledgement . . . and that the present laws, affecting your condition, are clearly unconstitutional. The fact that you have been treated, by common consent and common usage, as aliens and brutes, is not proof that such treatment is legal, but only shows the strength, the bitterness, and the blindness of prejudice.\\\"\\n—William Lloyd Garrison, \\\"To the Free People of Color of the United States,\\\" The Liberator, January 15, 1831\\nThe approach of William Lloyd Garrison and The Liberator can best be seen as\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}