{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15f840ff-7dcc-50e7-ac5d-1545126a4b7f","task_key":"default--test--4789","task_revision_id":"1","upstream_id":"4789","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Malcolm X\\\\'s \\\"The Ballot or the Bullet\\\" speech', 'Martin Luther King, Jr.\\\\'s \\\"I Have a Dream” speech', 'Abraham Lincoln\\\\'s \\\"Emancipation Proclamation\\\"', 'The Seneca Falls Declaration of Rights and Sentiments', 'Thomas Paine\\\\'s \\\"Common Sense\\\"', 'Nelson Mandela\\\\'s \\\"I Am Prepared to Die\\\" speech', 'Frederick Douglass\\\\'s \\\"What to the Slave is the Fourth of July?\\\"', 'Jonathan Edwards\\\\'s sermon \\\"Sinners in the Hands of an Angry God\\\"', 'John F. Kennedy\\\\'s \\\"Ich bin ein Berliner\\\" speech', 'The Declaration of Independence']\",\"question\":\"This question refers to the following information.\\nOn Being Brought from Africa to America\\n'Twas mercy brought me from my Pagan land,\\nTaught my benighted soul to understand\\nThat there's a God, that there's a Saviour too;\\nOnce I redemption neither sought nor knew.\\nSome view our sable race with scornful eye,\\n\\\"Their colour is a diabolic die.\\\"\\nRemember, Christians, Negroes, black as Cain,\\nMay be refin'd, and join th' angelic train.\\n—Phillis Wheatley, Poems on Various Subjects, Religious and Moral, 1773\\nThe point of Wheatley's poem can best be compared to which of the following?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}