{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"08554c8a-8931-55cf-995a-14b5ea94bf30","task_key":"default--test--4794","task_revision_id":"1","upstream_id":"4794","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Promote the industrial revolution across the continent.', \\\"Suppress all voices in government other than his own and control all aspects of his citizens' lives.\\\", 'Suppress all nationalist revolutions across the continent.', 'Suppress all speech but that of monarchs and the churches that they approve of.', 'Promote the spread of democratic revolutions.', 'Facilitate trade agreements between the Germanic States and other European nations.', 'Establish a unified German state under his control.', 'Establish control of European states by democracy in the tradition of the Enlightenment.', 'Promote religious freedom across the Germanic States.', 'Establish a single, universal language across all European states.']\",\"question\":\"This question refers to the following information.\\nRead the following governmental regulation.\\nPress Law\\nSo long as this decree shall remain in force no publication which appears in the form of daily issues, or as a serial not exceeding twenty sheets of printed matter, shall go to press in any state of the union without the previous knowledge and approval of the state officials. Writings which do not belong to one of the above-mentioned classes shall be treated according to the laws now in force, or which may be enacted, in the individual states of the union. . . . Each state of the union is responsible, not only to the state against which the offense is directly committed, but to the whole Confederation, for every publication appearing under its supervision in which the honor or security of other states is infringed or their constitution or administration attacked. . . .\\n—Carlsbad Resolutions adopted by the Germanic States, 1819\\nThe Carlsbad Resolutions were another of Metternich's schemes, like the Concert of Europe, which aimed to accomplish which of the following?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}