{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1aa826a-9579-5f3a-b286-c9c2d5c8b160","task_key":"all--test--f1aa826a-9579-5f3a-b286-c9c2d5c8b160","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Disagreement about the true nature of divinity', 'Disagreement about the role of the priesthood', 'Disagreement about the appropriate use of religious iconography', 'Disagreement about the best form of ascetic practice']\",\"question\":\"This question refers to the following information.\\nSource 1:\\n\\\"We proclaim Him also by our senses on all sides, and we sanctify the noblest sense, which is that of sight. The image is a memorial, just what words are to a listening ear. What a book is to the literate, that an image is to the illiterate. The image speaks to the sight as words to the ear; it brings us understanding.\\\"\\nJohn of Damascus, Apologia Against Those Who Decry Holy Images, ca. 730 C.E.\\nSource 2:\\n\\\"To make our confession short, we keep unchanged all the ecclesiastical traditions handed down to us, whether in writing or verbally, one of which is the making of pictorial representations, agreeable to the history of the preaching of the Gospel, a tradition useful in many respects, but especially in this, that so the incarnation of the Word of God is shown forth as real and not merely fantastic, for these have mutual indications and without doubt have also mutual significations.\\\"\\nDecree of the Second Council of Nicaea, 787 C.E.\\nThe late eighth-century religious debate in Byzantium that occasioned the writing of the passages is best understood in the context of which of the following?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}