{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65dafb7b-6507-574a-9be4-0141d348df9c","task_key":"all--test--65dafb7b-6507-574a-9be4-0141d348df9c","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The economic dominance of the Ottoman Empire', 'The influence of Middle Eastern religious beliefs on the peoples of South Asia', 'The lack of cooperation among major political groups in the High Middle Ages', 'Amiable cultural exchange between Europe and Asia']\",\"question\":\"This question refers to the following information.\\n\\\"Upon their arrival they were honorably and graciously received by the grand Khan, in a full assembly of his principal officers. When they drew nigh to his person, they paid their respects by prostrating themselves on the floor. He immediately commanded them to rise, and to relate to him the circumstances of their travels, with all that had taken place in their negotiation with his holiness the pope. To their narrative, which they gave in the regular order of events, and delivered in perspicuous language, he listened with attentive silence. The letters and the presents from pope Gregory were then laid before him, and, upon hearing the former read, he bestowed much commendation on the fidelity, the zeal, and the diligence of his ambassadors; and receiving with due reverence the oil from the holy sepulchre, he gave directions that it should be preserved with religious care. Upon his observing Marco Polo, and inquiring who he was, Nicolo made answer, This is your servant, and my son; upon which the grand Khan replied, \\\"He is welcome, and it pleases me much,\\\" and he caused him to be enrolled amongst his attendants of honor. And on account of their return he made a great feast and rejoicing; and as long as the said brothers and Marco remained in the court of the grand Khan, they were honored even above his own courtiers. Marco was held in high estimation and respect by all belonging to the court. He learnt in a short time and adopted the manners of the Tartars, and acquired a proficiency in four different languages, which he became qualified to read and write.\\\"\\nMarco Polo's Travels, ca. 1300 C.E.\\nThe encounter described in the account above illustrates which of the following?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}