{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"41f7f264-5659-5074-90a6-b04a7395fdd2","task_key":"all--test--41f7f264-5659-5074-90a6-b04a7395fdd2","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['were an expansionist empire geographically located between two other expansionists', 'rejected the central tenets of Islam', 'enslaved Christians, and turned them into Safavid fighters', 'Ismail proclaimed himself to be a Caliph']\",\"question\":\"This question refers to the following information.\\n\\\"Indeed, as both the fatwas of distinguished [scholars] who base their opinion on reason and tradition alike and the consensus of the Sunni community agree that the ancient obligation of extirpation, extermination, and expulsion of evil innovation must be the aim of our exalted aspiration, for \\\"Religious zeal is a victory for the Faith of God the Beneficent\\\"; then, in accordance with the words of the Prophet (Peace upon him!) \\\"Whosoever introduces evil innovation into our order must be expelled\\\" and \\\"Whosoever does aught against our order must be expelled,\\\" action has become necessary and exigent…\\\"\\nLetter from Ottoman Sultan Selim I to Safavid Shah Ismail I, 1514\\nThe Safavids drew the ire of Islamic empires such as the Ottoman and the Mughal Empires primarily because the Safavids\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}