{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"131a2b9f-d49c-5ee3-ad31-3d83da0bf714","task_key":"all--test--131a2b9f-d49c-5ee3-ad31-3d83da0bf714","task_revision_id":"1","upstream_id":"","short_description":"A state has a statute requiring all candidates for lieutenant governor to file a…","config":"all","split":"test","body":"{\"choices\":\"['it is constitutional under the Fifteenth Amendment.', 'it is necessary to further a compelling state interest.', \\\"the candidates's petition contained 2,000 false signatures.\\\", 'only one other candidate successfully petitioned to have his name placed on the ballot.']\",\"question\":\"A state has a statute requiring all candidates for lieutenant governor to file a petition with the signatures of 5,000 registered voters in order to have their names placed on the ballot. A candidate for lieutenant governor, did not receive the authorized number of signatures and did not have her name placed on the ballot. If the candidate brings an appropriate action challenging the constitutionality of the voter petition requirement, the best argument, if established, in support of the statute is that\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}