{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7efddd52-0bb5-53bc-a373-87c419b118aa","task_key":"all--test--7efddd52-0bb5-53bc-a373-87c419b118aa","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The effect liberal, young students would have on an impressionable faculty and the surrounding communities, where they would surely spread their message.', 'The dissemination of liberal ideas by the media and the sharing of nationalist or liberal ideas at the university in the classroom or in secret.', 'The dissolution of the union of German states that they had just created.', \\\"Support of Bonapartism, which was growing in strength prior to the passage of these Carlsbad Resolutions and preceeding Napoleon's invasion.\\\"]\",\"question\":\"This question refers to the following information.\\nRead the following governmental regulation.\\nPress Law\\nSo long as this decree shall remain in force no publication which appears in the form of daily issues, or as a serial not exceeding twenty sheets of printed matter, shall go to press in any state of the union without the previous knowledge and approval of the state officials. Writings which do not belong to one of the above-mentioned classes shall be treated according to the laws now in force, or which may be enacted, in the individual states of the union. . . . Each state of the union is responsible, not only to the state against which the offense is directly committed, but to the whole Confederation, for every publication appearing under its supervision in which the honor or security of other states is infringed or their constitution or administration attacked. . . .\\n—Carlsbad Resolutions adopted by the Germanic States, 1819\\nAccording to the above-mentioned decrees, the German confederated states most feared which of the following?\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}