{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9fb047c5-de9a-5d71-9d2c-59150a0a7cfc","task_key":"default--test--4975","task_revision_id":"1","upstream_id":"4975","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The United States Senate approved it without any reservations.', 'The United States Senate rejected it because it treated Germany too leniently.', 'The United States Senate rejected it because it required increased American involvement in European affairs.', 'The United States Senate approved it, with reservations concerning the League of Nations.', 'The United States Senate approved it because it required decreased American involvement in European affairs.', 'The United States Senate approved it, with reservations concerning the division of Eastern Europe.', 'It was voted down by the United States Senate because it did not include the provision for an impartial neutrality.', 'It was unanimously approved by the United States Senate.', 'The United States Senate rejected it because it did not address the issue of neutrality.', 'It was never voted on by the United States Senate.']\",\"question\":\"This question refers to the following information.\\n\\\"Whereas the laws and treaties of the United States, without interfering with the free expression of opinion and sympathy, or with the commercial manufacture or sale of arms or munitions of war, nevertheless impose upon all persons who may be within their territory and jurisdiction the duty of an impartial neutrality during the existence of the contest; And Whereas it is the duty of a neutral government not to permit or suffer the making of its waters subservient to the purposes of war;\\n\\\"Now, Therefore, I, Woodrow Wilson, President of the United States of America, in order to preserve the neutrality of the United States…do hereby declare and proclaim….\\n\\\"That the statutes and the treaties of the United States and the law of nations alike require that no person, within the territory and jurisdiction of the United States, shall take part, directly or indirectly, in the said wars, but shall remain at peace with all of the said belligerents, and shall maintain a strict and impartial neutrality….\\\"\\nWoodrow Wilson, 1914\\nWhich of the following statements about the Treaty of Versailles is true?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}