{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"049fc1c9-d813-5dc3-b9d5-12489328291e","task_key":"default--test--4896","task_revision_id":"1","upstream_id":"4896","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['resulted in the isolation of Russian territories, while the Mongol administration in China facilitated international trade and communication', 'was only tolerant of Orthodox Christianity, while the Mongol administration in China favored many religious traditions', 'relied heavily on tribute from the Russian principalities, while the Mongol administration in China emulated a centralized Chinese bureaucratic state', 'granted autonomy to local rulers, while the Mongol administration in China imposed direct control', 'imposed Islamic law in its domains, while the Mongol administration in China allowed for diverse religious practices', 'was viewed favorably by its subjects, while the Mongol administration in China encountered immense domestic unrest', 'led to the decline of cities, while the Mongol administration in China encouraged urban development', 'lacked a well-defined administrative structure, while the Mongol administration in China established a robust bureaucratic system', 'stimulated Russian export of trade goods to China, while the Mongol administration in China intentionally curtailed Chinese economic activity', 'discouraged inter-cultural exchanges, while the Mongol administration in China promoted cultural integration']\",\"question\":\"This question refers to the following information.\\n\\\"The Chinese are all infidels: they worship images, and burn their dead just like the [Hindus]. The King of China is a Tartar, and one of the descendants of [Ghenghis] Khan…In all the Chinese provinces, there is a town for the [Muslims], and in this they reside. They also have cells, colleges, and mosques, and are made much of by the Kings of China…\\nWhen we approached this city [of Hangzhou] we were met by its judge, the [elders] of Islamism, and the great merchants. The [Muslims] are exceedingly numerous here. This whole city is surrounded by a wall: each of the six [districts of Hangzhou] is also surrounded by a wall. In the first reside the guards, with their commander. I was told that, in the muster-rolls, these amount to twelve thousand…In the second division are the Jews, Christians, and the Turks who worship the sun: these are numerous, their number is not known: and theirs is the most beautiful city. Their streets are well disposed, and their great men are exceedingly wealthy. There are in the city a great number of [Muslims], with some of whom I resided for fifteen days; and was treated most [honorably]…\\\"\\nIbn Battuta, Voyages, 1332–1346 C.E.\\nCompared to the observations expressed in the excerpt, Mongol administration of its Russian domains in the period 600 C.E. to 1450 C.E. differed in that it\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}