{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"95c22dba-70e7-5998-83f6-aa45f3576df3","task_key":"all--test--95c22dba-70e7-5998-83f6-aa45f3576df3","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Debates over religious freedom', 'Debates over social justice', 'Debates over immigration', 'Debates over freedom of expression']\",\"question\":\"This question refers to the following information.\\nOn Being Brought from Africa to America\\n'Twas mercy brought me from my Pagan land,\\nTaught my benighted soul to understand\\nThat there's a God, that there's a Saviour too;\\nOnce I redemption neither sought nor knew.\\nSome view our sable race with scornful eye,\\n\\\"Their colour is a diabolic die.\\\"\\nRemember, Christians, Negroes, black as Cain,\\nMay be refin'd, and join th' angelic train.\\n—Phillis Wheatley, Poems on Various Subjects, Religious and Moral, 1773\\nThe sentiments expressed in Wheatley's poem most directly reflect which of the following continuities in U.S. history?\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}