{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"edb2ac42-3e86-5bff-a14c-c0520c753bfd","task_key":"all--test--edb2ac42-3e86-5bff-a14c-c0520c753bfd","task_revision_id":"1","upstream_id":"","short_description":"Which of the following is an accurate statement about the leadership of the two…","config":"all","split":"test","body":"{\"choices\":\"['The national party organization controls all facets of party activity throughout the country.', 'Party leadership is dispersed among numerous officials at the national, state, and local levels.', \\\"The highest elected official in each national party directs that party's activities and operations all the way down to the state and local levels.\\\", \\\"All major decisions concerning party activities are made during each party's presidential nomination conventions.\\\"]\",\"question\":\"Which of the following is an accurate statement about the leadership of the two major political parties?\",\"subject\":\"high_school_government_and_politics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}