{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c0488e16-21bc-5bb6-ba5b-3ea2756e9f3f","task_key":"default--test--5054","task_revision_id":"1","upstream_id":"5054","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Soviet Union saw steady growth in agriculture but industrial production fell behind.', 'The Soviet Union saw temporary growth in industry, followed by a rapid decline.', 'The Soviet Union saw steady growth in industry and a decline in agriculture.', 'The Soviet Union saw steady decline in both industry and agriculture.', 'The Soviet Union saw a decline in both industry and agriculture, but then rebounded.', \\\"The Soviet Union's industrial and agricultural sectors remained stagnant.\\\", 'The Soviet Union saw a rapid growth in both industry and agriculture, followed by a catastrophic collapse.', 'The Soviet Union saw steady growth in both industry and agriculture.', 'The Soviet Union saw a decline in industry but steady growth in agriculture.', \\\"The Soviet Union became the world's leading industrial power.\\\"]\",\"question\":\"This question refers to the following information.\\nThe following excerpt is from a speech by Vladimir I. Lenin.\\nThe independence of our country cannot be upheld unless we have an adequate industrial basis for defense. And such an industrial basis cannot be created if our industry is not more highly developed technically. That is why a fast rate of development of our industry is necessary and imperative. We cannot go on indefinitely, that is, for too long a period, basing the Soviet regime and socialist construction on two different foundations, the foundation of the most large-scale and united socialist industry and the foundation of the most scattered and backward, small commodity economy of the peasants. We must gradually, but systematically and persistently, place our agriculture on a new technical basis, the basis of large-scale production, and bring it up to the level of socialist industry. Either we accomplish this task—in which case the final victory of socialism in our country will be assured, or we turn away from it and do not accomplish it—in which case a return to capitalism may become inevitable.\\n—Vladimir I. Lenin Industrialization of the Country and the Right Deviation in the C.P.S.U., November 19,1928\\nWhat was the result of Stalin's attempt to fix this problem?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}