{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6bd2453b-6dc0-5644-a572-0c36c7548500","task_key":"default--test--4727","task_revision_id":"1","upstream_id":"4727","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['That most Muslims during the crusading era were driven by economic gain rather than religious fervor', 'That Muslims during the crusading era were primarily focused on peaceful co-existence with Christians', 'That the majority of Muslims during the crusading era were more interested in scholarly pursuits than warfare', 'That the majority of Muslims were more peaceful than Christians during the crusading era', 'That many Muslims during the crusading era were driven by Islamic faith to be inherently violent', 'That most Muslims were interested only in protecting their own territory during the crusading era', 'That the ideology of holy war was taken seriously by Muslim elites during the crusading era', 'That the ideology of holy war was dismissed by Muslim elites during the crusading era', 'That the majority of Muslims were indifferent towards the ideology of holy war during the crusading era', 'That most Muslims were interested in converting Christians to Islam during the crusading era']\",\"question\":\"This question refers to the following information.\\nWhile some theorists in the Middle Ages argued that the jihad was a defensive war… most authorities held that the obligation of jihad did not lapse until all the world was brought under the sway of Islam. The Bahr [al-Fava'id, or \\\"Sea of Precious Virtues,\\\" written in the 1150s or 1160s] insists that the first duty of a Muslim ruler is to prosecute the jihad and bring about the victory of Islam, and if he does not do so and he makes peace with the infidel, that ruler would be better dead than alive, for he would be corrupting the world.\\nRobert Irwin, \\\"Islam and the Crusades,\\\" 1995\\nIt is strange how the Christians round Mount Lebanon, when they see any Muslim hermits, bring them food and treat them kindly, saying that these men are dedicated to the Great and Glorious God and that they should therefore share with them. Likewise, not one Christian merchant was stopped or hindered in Muslim territories.\\nIbn Jubayr, Muslim scholar, traveling to Mecca and Jerusalem, ca. 1185\\nThe first passage tends to support which of the following traditional historical assumptions?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}