{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"513ee1d6-3f22-537b-8a97-0f3f4383ecff","task_key":"default--test--4743","task_revision_id":"1","upstream_id":"4743","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Watergate scandal and subsequent resignation of President Nixon in 1974', 'The legal battles over the Affordable Care Act in the early 2010s', 'The trial of the Rosenbergs for espionage during the Cold War', 'The Supreme Court decision in Roe v. Wade in 1973', 'The Brown v. Board of Education Supreme Court decision of 1954', 'The trial of the accused Haymarket Square bombers in 1886', 'The impeachment trial of President Andrew Johnson in 1868', \\\"Abraham Lincoln's suspension of habeas corpus during the Civil War\\\", 'Government efforts to prevent the publication of the Pentagon Papers in 1971', 'The Supreme Court decision in Citizens United v. FEC in 2010']\",\"question\":\"This question refers to the following information.\\nIt is natural, it is a privilege, I will go farther, it is a right, which all free men claim, that they are entitled to complain when they are hurt. They have a right publicly to remonstrate against the abuses of power in the strongest terms, to put their neighbors upon their guard against the craft or open violence of men in authority, and to assert with courage the sense they have of the blessings of liberty, the value they put upon it, and their resolution at all hazards to preserve it as one of the greatest blessings heaven can bestow.… But to conclude: The question before the Court and you, Gentlemen of the jury, is not of small or private concern. It is not the cause of one poor printer, nor of New York alone, which you are now trying. No! It may in its consequence affect every free man that lives under a British government on the main of America. It is the best cause. It is the cause of liberty. And I make no doubt but your upright conduct this day will not only entitle you to the love and esteem of your fellow citizens, but every man who prefers freedom to a life of slavery will bless and honor you as men who have baffled the attempt of tyranny, and by an impartial and uncorrupt verdict have laid a noble foundation for securing to ourselves, our posterity, and our neighbors, that to which nature and the laws of our country have given us a right to liberty of both exposing and opposing arbitrary power (in these parts of the world at least) by speaking and writing truth.\\n—Andrew Hamilton, concluding argument, libel trial of newspaper editor\\nJohn Peter Zenger, August 4, 1735\\nThe Zenger Case can best be compared to which of the following?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}