{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3ad16982-ffd4-556c-9783-00c344a66114","task_key":"default--test--4920","task_revision_id":"1","upstream_id":"4920","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['had a more advanced military than the Ottoman and Mughal Empires', 'were an expansionist empire geographically located between two other expansionists', 'promoted polytheism, which went against the beliefs of the Ottoman and Mughal Empires', 'were accused of blasphemy by the Ottoman and Mughal Empires', 'Ismail proclaimed himself to be a Caliph', 'refused to pay tributes to the Ottoman and Mughal Empires', 'believed in a different sect of Islam', 'were allies with the enemies of the Ottoman and Mughal Empires', 'rejected the central tenets of Islam', 'enslaved Christians, and turned them into Safavid fighters']\",\"question\":\"This question refers to the following information.\\n\\\"Indeed, as both the fatwas of distinguished [scholars] who base their opinion on reason and tradition alike and the consensus of the Sunni community agree that the ancient obligation of extirpation, extermination, and expulsion of evil innovation must be the aim of our exalted aspiration, for \\\"Religious zeal is a victory for the Faith of God the Beneficent\\\"; then, in accordance with the words of the Prophet (Peace upon him!) \\\"Whosoever introduces evil innovation into our order must be expelled\\\" and \\\"Whosoever does aught against our order must be expelled,\\\" action has become necessary and exigent…\\\"\\nLetter from Ottoman Sultan Selim I to Safavid Shah Ismail I, 1514\\nThe Safavids drew the ire of Islamic empires such as the Ottoman and the Mughal Empires primarily because the Safavids\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}