{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b25b4844-9438-517e-afdb-aabbba70584b","task_key":"default--test--4888","task_revision_id":"1","upstream_id":"4888","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['a restructuring of the socialist society of the Soviet Union was not an urgent matter', 'a restructuring of the socialist society of the Soviet Union was being forced by external factors', 'a restructuring of the socialist society of the Soviet Union would exacerbate the current situation', 'a restructuring of the socialist society of the Soviet Union would have no significant impact on the current situation', 'a restructuring of socialist society was necessary in order to avoid the collapse of the Soviet Union', 'a restructuring of the socialist society of the Soviet Union was primarily to appease the masses', 'a restructuring of the socialist society of the Soviet Union would lead to a collapse of the political system', 'a restructuring of the socialist society of the Soviet Union was necessary in order to avoid a crisis', 'a restructuring of the socialist society of the Soviet Union was inevitable', 'a restructuring of the socialist society of the Soviet Union would lead to a strengthened economic system']\",\"question\":\"This question refers to the following information.\\nPerestroika [Restructuring] is an urgent necessity arising from the profound processes of development in our socialist society. This society is ripe for change. It has long been yearning for it. Any delay in beginning perestroika could have led to an exacerbated internal situation in the near future, which, to put it bluntly, would have been fraught with serious social, economic, and political crises.\\nMikhail Gorbachev, Perestroika: New Thinking for Our Country and the World, 1987\\nFrom the passage, one may infer that Gorbachev argued that\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}