{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"022b06e8-fca8-5d55-b511-9f4ba340e686","task_key":"all--test--022b06e8-fca8-5d55-b511-9f4ba340e686","task_revision_id":"1","upstream_id":"","short_description":"The book announced an insane world of dehumanization through terror in which the…","config":"all","split":"test","body":"{\"choices\":\"[\\\"E.M. Forster's A Passage to India\\\", \\\"Thomas Pynchon's V.\\\", \\\"George Orwell's 1984\\\", \\\"Flannery O'Connor's The Violent Bear It Away\\\"]\",\"question\":\"The book announced an insane world of dehumanization through terror in which the individual was systematically obliterated by an all-powerful elite. Its key phrases‚ÄîBig Brother, doublethink, Newspeak, the Ministry of Peace (devoted to war), the Ministry of Truth (devoted to lies), the Ministry of Love (devoted to torture)‚Äîburned their way at once into the modern consciousness. The passage above discusses\",\"subject\":\"miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}