{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b1cfa4bf-1c9e-59d2-908d-befdac486e2f","task_key":"all--test--b1cfa4bf-1c9e-59d2-908d-befdac486e2f","task_revision_id":"1","upstream_id":"","short_description":"Which of the following best describes the fate of most popular third-party…","config":"all","split":"test","body":"{\"choices\":\"['They displace one of the two major parties and become major parties themselves.', 'They are ultimately abandoned by the public because their politics are perceived as too radical.', 'Their supporters become frustrated and withdraw from the political process.', \\\"They disintegrate when one or both of the major parties adopt the third party's goals.\\\"]\",\"question\":\"Which of the following best describes the fate of most popular third-party movements?\",\"subject\":\"high_school_government_and_politics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}