{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7424ee9-8c47-53d5-aa5e-9c31c2d38881","task_key":"default--test--4983","task_revision_id":"1","upstream_id":"4983","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The natural rights theory of John Locke', 'British ideas about social hierarchy', 'The Romantic literary movement', 'The First Great Awakening', 'The Transcendentalist movement', 'The American Revolution', 'The Second Great Awakening', 'Eighteenth century scientific racism', 'The Enlightenment philosophy', 'The Abolitionist movement']\",\"question\":\"This question refers to the following information.\\nOn Being Brought from Africa to America\\n'Twas mercy brought me from my Pagan land,\\nTaught my benighted soul to understand\\nThat there's a God, that there's a Saviour too;\\nOnce I redemption neither sought nor knew.\\nSome view our sable race with scornful eye,\\n\\\"Their colour is a diabolic die.\\\"\\nRemember, Christians, Negroes, black as Cain,\\nMay be refin'd, and join th' angelic train.\\n—Phillis Wheatley, Poems on Various Subjects, Religious and Moral, 1773\\nThe ideas expressed in Phillis Wheatley's poem most directly reveal the influence of which of the following?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}