{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0f05c555-d4e9-50a3-8858-6f5ccdeb1f86","task_key":"default--test--4713","task_revision_id":"1","upstream_id":"4713","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The United States annexed it after a revolt by American settlers living in the territory.', 'The United States bought it from the Native Americans who lived there.', 'The territory was divided between the United States and Canada after a vote by residents.', 'U.S. settlers were the first to arrive in the region; they claimed it for their country.', 'The United States claimed it after discovering it during an exploration.', 'The United States won it from Mexico in a war.', 'The United States inherited it from Spain as part of the Louisiana Purchase.', 'Great Britain ceded it to the United States as part of a negotiated treaty.', 'The United States was granted the territory in a postwar treaty with France.', 'The territory was gifted to the United States by Russia.']\",\"question\":\"This question refers to the following information.\\n\\\"The far-reaching, the boundless future will be the era of American greatness. In its magnificent domain of space and time, the nation of many nations is destined to manifest to mankind the excellence of divine principles; to establish on earth the noblest temple ever dedicated to the worship of the Most High—the Sacred and the True. Its floor shall be a hemisphere—its roof the firmament of the star-studded heavens, and its congregation a Union of many Republics, comprising hundreds of happy millions, calling, owning no man master, but governed by God's natural and moral law of equality, the law of brotherhood—of 'peace and good will amongst men.'\\\"\\nJohn L. O'Sullivan, \\\"The Great Nation of Futurity,\\\" 1839\\nBy what means did the United States take possession of the Oregon Territory?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}