{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"12b04b63-166e-5d6b-9abc-d44062c95712","task_key":"default--test--4788","task_revision_id":"1","upstream_id":"4788","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The number of pioneers headed for the Oregon territory increased while the number headed for California decreased.', 'The first great wave of migration ended, and the number of migrants remained extremely low until after the Civil War.', 'The number of pioneers headed for the Oregon territory decreased while the number headed for California greatly increased.', 'Large numbers of Native Americans began to migrate eastward, in response to the increasing westward migration.', 'The number of pioneers headed for both the Oregon territory and California remained the same.', 'The migration remained steady, without any significant changes in the direction or volume of pioneers.', 'For the first time, pioneers began to settle areas west of the Mississippi River.', 'The Californian Gold Rush caused a decrease in migration, as people chose to stay and mine for gold.', 'Most pioneers chose to settle in the East due to the hardships of the West.', 'Large numbers of free blacks, unwelcome in the East, began to resettle in the West.']\",\"question\":\"This question refers to the following information.\\n\\\"The far-reaching, the boundless future will be the era of American greatness. In its magnificent domain of space and time, the nation of many nations is destined to manifest to mankind the excellence of divine principles; to establish on earth the noblest temple ever dedicated to the worship of the Most High—the Sacred and the True. Its floor shall be a hemisphere—its roof the firmament of the star-studded heavens, and its congregation a Union of many Republics, comprising hundreds of happy millions, calling, owning no man master, but governed by God's natural and moral law of equality, the law of brotherhood—of 'peace and good will amongst men.'\\\"\\nJohn L. O'Sullivan, \\\"The Great Nation of Futurity,\\\" 1839\\nWhich of the following changes in westward migration is most likely to have occurred in the 1840s?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}