{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"349ceb7f-e5aa-5f61-bbbe-11be6002d706","task_key":"default--test--4885","task_revision_id":"1","upstream_id":"4885","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['corporations were entitled to the same protections guaranteed to individuals under the Fourteenth Amendment', 'the Fugitive Slave Act was unconstitutional', 'women had the right to vote under the United States Constitution', 'separate but equal facilities for people of different races was constitutional', 'every state had the right to secede from the Union', 'the abolitionist movement was a threat to the United States Constitution', 'school prayer violated the principle of \\\"separation of church and state\\\"', 'the Second Amendment allowed individuals to keep and bear arms', \\\"slavery was a state's rights issue and should be decided by popular sovereignty\\\", 'Congress had no right to regulate slavery in United States territories']\",\"question\":\"This question refers to the following information.\\n\\\"In one view the slaveholders have a decided advantage over all opposition. It is well to notice this advantage—the advantage of complete organization. They are organized; and yet were not at the pains of creating their organizations. The State governments, where the system of slavery exists, are complete slavery organizations. The church organizations in those States are equally at the service of slavery; while the Federal Government, with its army and navy, from the chief magistracy in Washington, to the Supreme Court, and thence to the chief marshalship at New York, is pledged to support, defend, and propagate the crying curse of human bondage. The pen, the purse, and the sword, are united against the simple truth, preached by humble men in obscure places.\\\"\\nFrederick Douglass, 1857\\nIn his opinion on the case Dred Scott v. Sandford, Chief Justice Roger Taney upheld the sentiment above by stating that\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}