{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5d99c4cc-17b8-5a8f-b442-015e94a37269","task_key":"default--test--4894","task_revision_id":"1","upstream_id":"4894","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Technological developments were hindered due to religious beliefs.', 'Technological developments spread across trade routes.', 'The empires of steppe nomads united trade links across Eurasia.', 'Religious tolerance was not practiced in major empires.', 'New religious traditions overturned prior lasting religious beliefs.', 'The Chinese empire was isolated and had little contact with other civilizations.', 'Trade was primarily conducted within individual civilizations and not between them.', 'The spread of Islam was primarily through military conquest.', 'The Chinese empire was primarily Jewish and Christian during this period.', 'Long-distance contact between civilizations stagnated.']\",\"question\":\"This question refers to the following information.\\n\\\"The Chinese are all infidels: they worship images, and burn their dead just like the [Hindus]. The King of China is a Tartar, and one of the descendants of [Ghenghis] Khan…In all the Chinese provinces, there is a town for the [Muslims], and in this they reside. They also have cells, colleges, and mosques, and are made much of by the Kings of China…\\nWhen we approached this city [of Hangzhou] we were met by its judge, the [elders] of Islamism, and the great merchants. The [Muslims] are exceedingly numerous here. This whole city is surrounded by a wall: each of the six [districts of Hangzhou] is also surrounded by a wall. In the first reside the guards, with their commander. I was told that, in the muster-rolls, these amount to twelve thousand…In the second division are the Jews, Christians, and the Turks who worship the sun: these are numerous, their number is not known: and theirs is the most beautiful city. Their streets are well disposed, and their great men are exceedingly wealthy. There are in the city a great number of [Muslims], with some of whom I resided for fifteen days; and was treated most [honorably]…\\\"\\nIbn Battuta, Voyages, 1332–1346 C.E.\\nWhich of the following conclusions about the period 600 C.E.–1450 C.E. is most directly supported by the passage?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}