{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8aea8c77-7bd8-5bc3-8418-5b4eb50e1e37","task_key":"default--test--4892","task_revision_id":"1","upstream_id":"4892","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['John Stuart Mill', 'Napoleon Bonaparte', 'Simon Bolivar', 'Queen Victoria', 'William Frederick I', 'Pope Pius IX', 'Otto von Bismarck', 'Giuseppe Mazzini', 'Karl Marx', 'Klemens Von Metternich']\",\"question\":\"This question refers to the following information.\\nRead the following governmental regulation.\\nPress Law\\nSo long as this decree shall remain in force no publication which appears in the form of daily issues, or as a serial not exceeding twenty sheets of printed matter, shall go to press in any state of the union without the previous knowledge and approval of the state officials. Writings which do not belong to one of the above-mentioned classes shall be treated according to the laws now in force, or which may be enacted, in the individual states of the union. . . . Each state of the union is responsible, not only to the state against which the offense is directly committed, but to the whole Confederation, for every publication appearing under its supervision in which the honor or security of other states is infringed or their constitution or administration attacked. . . .\\n—Carlsbad Resolutions adopted by the Germanic States, 1819\\nWhich of the following nineteenth-century figures would have been the LEAST likely to oppose those liberals described in the above decrees?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}