{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"babcc91a-97c6-5078-8619-1fabae44b8a9","task_key":"default--test--4695","task_revision_id":"1","upstream_id":"4695","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Treaty of Tordesillas (1494)', 'Lateran Treaty (1929)', 'Treaty of Versailles (1919)', 'Kellogg-Briand Pact (1928)', 'Sykes-Picot Agreement (1916)', 'Treaty of Ghent (1814)', 'Treaty of Brest-Litovsk (1918)', 'Munich Agreement (1938)', 'Treaty of Tilsit (1807)', 'Triple Entente (1907)']\",\"question\":\"This question refers to the following information.\\n\\\"The Government of the German Reich and The Government of the Union of Soviet Socialist Republics desirous of strengthening the cause of peace between Germany and the U.S.S.R., and proceeding from the fundamental provisions of the Neutrality Agreement concluded in April, 1926 between Germany and the U.S.S.R., have reached the following Agreement:\\nArticle I. Both High Contracting Parties obligate themselves to desist from any act of violence, any aggressive action, and any attack on each other, either individually or jointly with other Powers.\\nArticle II. Should one of the High Contracting Parties become the object of belligerent action by a third Power, the other High Contracting Party shall in no manner lend its support to this third Power.\\nArticle III. The Governments of the two High Contracting Parties shall in the future maintain continual contact with one another for the purpose of consultation in order to exchange information on problems affecting their common interests.\\nArticle IV. Should disputes or conflicts arise between the High Contracting Parties shall participate in any grouping of Powers whatsoever that is directly or indirectly aimed at the other party.\\nArticle V. Should disputes or conflicts arise between the High Contracting Parties over problems of one kind or another, both parties shall settle these disputes or conflicts exclusively through friendly exchange of opinion or, if necessary, through the establishment of arbitration commissions.\\\"\\nMolotov-Ribbentrop Pact, 1939\\nThe agreement described in the text is most similar to the\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}