{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c86346c4-57b9-5d88-ae05-7b4c34f427b1","task_key":"all--test--c86346c4-57b9-5d88-ae05-7b4c34f427b1","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"[\\\"the clergy's monopoly on academic scholarship in eighteenth-century Britain\\\", 'the hatred of all things Roman by British scholars in the eighteenth century', 'the spread of religious skepticism among the educated elite of Britain in the eighteenth century', 'the lack of sources available to the eighteenth-century scholar for the study of ancient Roman civilization']\",\"question\":\"This question refers to the following information.\\nRead the the following quotation to answer questions.\\nThe various modes of worship which prevailed in the Roman world were all considered by the people as equally true; by the philosopher as equally false; and by the magistrate as equally useful.\\nEdward Gibbon, The Decline and Fall of the Roman Empire, 1776–1788\\nThe interpretation of the state of religious belief in ancient Rome by the eighteenth-century English historian Edward Gibbon might be offered as evidence for\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}