{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b7826eda-704c-59f5-bc04-3bf16cfbe742","task_key":"all--test--b7826eda-704c-59f5-bc04-3bf16cfbe742","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"[\\\"Justice Frank Murphy's dissent in the Supreme Court case, Korematsu v. United States in 1944.\\\", \\\"U.S. Army lawyer Joseph Welsh's opposition to Senator Joseph McCarthy in the Army-McCarthy hearings in 1954.\\\", \\\"Governor Orval Faubus's response to the steps taken by President Dwight Eisenhower to resolve the Little Rock crisis in 1957.\\\", \\\"John Lewis's endorsement of the Voting Rights Act in 1965.\\\"]\",\"question\":\"This question refers to the following information.\\n\\\"The power . . . given to the commanding officer over all the people of each district is that of an absolute monarch. His mere will is to take the place of all law. . . . It reduces the whole population of the ten states—all persons, of every color, sex, and condition, and every stranger within their limits—to the most abject and degrading slavery.\\\"\\nThe political sentiment of the veto message above is most similar to which of the following political positions taken in the twentieth century?\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}