{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27354908-057e-5c7a-9e5b-425c6cd9b0f3","task_key":"all--test--27354908-057e-5c7a-9e5b-425c6cd9b0f3","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['showing that state-level antagonisms in the Middle East during the crusading era were not always reflected in personal or economic life.', 'showing how Middle Eastern Muslims during the crusading era did not take religious obligations as seriously as Christians did.', 'showing how Muslims were on the whole kinder than Christians in the Middle East during the crusading era.', 'showing how, in the Middle East during the crusading era, economic factors trumped all other considerations.']\",\"question\":\"This question refers to the following information.\\nWhile some theorists in the Middle Ages argued that the jihad was a defensive war… most authorities held that the obligation of jihad did not lapse until all the world was brought under the sway of Islam. The Bahr [al-Fava'id, or \\\"Sea of Precious Virtues,\\\" written in the 1150s or 1160s] insists that the first duty of a Muslim ruler is to prosecute the jihad and bring about the victory of Islam, and if he does not do so and he makes peace with the infidel, that ruler would be better dead than alive, for he would be corrupting the world.\\nRobert Irwin, \\\"Islam and the Crusades,\\\" 1995\\nIt is strange how the Christians round Mount Lebanon, when they see any Muslim hermits, bring them food and treat them kindly, saying that these men are dedicated to the Great and Glorious God and that they should therefore share with them. Likewise, not one Christian merchant was stopped or hindered in Muslim territories.\\nIbn Jubayr, Muslim scholar, traveling to Mecca and Jerusalem, ca. 1185\\nThe second passage undermines traditional historical assumptions by\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}