{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"adb4b3f4-851a-5bee-a74e-2c02471b82f3","task_key":"default--test--4923","task_revision_id":"1","upstream_id":"4923","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Chinese government banned all foreign trade to protect domestic industries.', 'The Chinese government restricted foreign merchants to specific sites in the coastal cities.', 'The Chinese government required foreign merchants to adopt Chinese customs and practices.', 'The Chinese government endorsed Chinese merchants to conduct trade missions abroad.', 'The Chinese government abandoned Confucian principles to allow merchants a greater participation in local government.', 'The Chinese government completely eliminated all restrictions on foreign trade.', 'The Chinese government allowed unrestricted trade throughout the country.', 'The Chinese government gave foreign merchants prominent roles in local government.', 'The Chinese government imposed heavy taxes on all foreign merchants.', 'The Chinese government monopolized the production and sale of key resources, such as salt and iron.']\",\"question\":\"This question refers to the following information.\\n\\\"The Chinese are all infidels: they worship images, and burn their dead just like the [Hindus]. The King of China is a Tartar, and one of the descendants of [Ghenghis] Khan…In all the Chinese provinces, there is a town for the [Muslims], and in this they reside. They also have cells, colleges, and mosques, and are made much of by the Kings of China…\\nWhen we approached this city [of Hangzhou] we were met by its judge, the [elders] of Islamism, and the great merchants. The [Muslims] are exceedingly numerous here. This whole city is surrounded by a wall: each of the six [districts of Hangzhou] is also surrounded by a wall. In the first reside the guards, with their commander. I was told that, in the muster-rolls, these amount to twelve thousand…In the second division are the Jews, Christians, and the Turks who worship the sun: these are numerous, their number is not known: and theirs is the most beautiful city. Their streets are well disposed, and their great men are exceedingly wealthy. There are in the city a great number of [Muslims], with some of whom I resided for fifteen days; and was treated most [honorably]…\\\"\\nIbn Battuta, Voyages, 1332–1346 C.E.\\nWhich of the following changes to Chinese policies regarding trade occurred under the Ming Dynasty of China?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}