{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2864ac6d-e4d8-5b25-95cd-73e0b01f2209","task_key":"default--test--5019","task_revision_id":"1","upstream_id":"5019","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"President George W. Bush's implementation of the Patriot Act in 2001.\\\", \\\"The Black Panther Party's opposition to police brutality in the 1960s.\\\", \\\"President Richard Nixon's response to the Watergate scandal in 1974.\\\", \\\"Governor Orval Faubus's response to the steps taken by President Dwight Eisenhower to resolve the Little Rock crisis in 1957.\\\", \\\"Senator Bernie Sanders's stance on income inequality in 2016.\\\", \\\"John Lewis's endorsement of the Voting Rights Act in 1965.\\\", \\\"President Franklin D. Roosevelt's decision to intern Japanese-Americans during World War II in 1942.\\\", \\\"U.S. Army lawyer Joseph Welsh's opposition to Senator Joseph McCarthy in the Army-McCarthy hearings in 1954.\\\", \\\"Justice Frank Murphy's dissent in the Supreme Court case, Korematsu v. United States in 1944.\\\", \\\"Martin Luther King Jr.'s criticism of the Vietnam War in 1967.\\\"]\",\"question\":\"This question refers to the following information.\\n\\\"The power . . . given to the commanding officer over all the people of each district is that of an absolute monarch. His mere will is to take the place of all law. . . . It reduces the whole population of the ten states—all persons, of every color, sex, and condition, and every stranger within their limits—to the most abject and degrading slavery.\\\"\\nThe political sentiment of the veto message above is most similar to which of the following political positions taken in the twentieth century?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}