{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d26b5c0b-b49e-53d4-95d0-57a0e0a6baca","task_key":"default--test--4881","task_revision_id":"1","upstream_id":"4881","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the creation of a strong executive office to lead the national government', 'the establishment of the First National Bank', 'the empowerment of Congress to collect taxes', 'the empowerment of Congress to regulate commerce', 'the creation of a bicameral legislature', 'the provision for land sales in the Northwest that would benefit the entire nation', 'the abolition of slavery', 'the creation of a national army', 'the introduction of a national currency', 'the establishment of a national judiciary system']\",\"question\":\"This question refers to the following information.\\n\\\"Permit us, then, earnestly to recommend these articles to the immediate and dispassionate attention of the legislatures of the respective states. Let them be candidly reviewed under a sense of the difficulty of combining in one system the various sentiments and interests of a continent divided into so many sovereign and independent communities, under a conviction of the absolute necessity of uniting all our councils and all our strength, to maintain and defend our common liberties….\\\"\\nJournals of the Continental Congress, 1777\\nThe most notable achievement of the United States under the Articles of Confederation was\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}