{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2d14e738-c815-549f-88e3-b7b68a23e38c","task_key":"default--test--4697","task_revision_id":"1","upstream_id":"4697","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The attempt of a religious group to proselytize to those following another faith', 'The religious motivations behind the Crusades', 'The suppression of minority religions in a dominant religious society', 'The influence of religion on interaction between cultures', 'The evolution of religious rituals and customs over time', 'The syncretic fusion of different religious traditions', 'The political manipulation of religious beliefs', 'The spread of Islam through peaceful means']\",\"question\":\"This question refers to the following information.\\nWhile some theorists in the Middle Ages argued that the jihad was a defensive war… most authorities held that the obligation of jihad did not lapse until all the world was brought under the sway of Islam. The Bahr [al-Fava'id, or \\\"Sea of Precious Virtues,\\\" written in the 1150s or 1160s] insists that the first duty of a Muslim ruler is to prosecute the jihad and bring about the victory of Islam, and if he does not do so and he makes peace with the infidel, that ruler would be better dead than alive, for he would be corrupting the world.\\nRobert Irwin, \\\"Islam and the Crusades,\\\" 1995\\nIt is strange how the Christians round Mount Lebanon, when they see any Muslim hermits, bring them food and treat them kindly, saying that these men are dedicated to the Great and Glorious God and that they should therefore share with them. Likewise, not one Christian merchant was stopped or hindered in Muslim territories.\\nIbn Jubayr, Muslim scholar, traveling to Mecca and Jerusalem, ca. 1185\\nThese two passages are best understood in the context of which of the following?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}