{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b8b76aa2-6bf9-55a3-8543-d508b9fd7769","task_key":"default--test--4981","task_revision_id":"1","upstream_id":"4981","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the increase in immigration, the growth of labor unions, and the rise of the feminist movement.', 'the baby boom, economic growth, and a rapid expansion of higher education.', 'the proliferation of personal computer technologies, the rise of Christian fundamentalism, and an increase in student apathy.', 'the escalation of the Vietnam War, the growth of the peace movement, and the assassination of key political figures.', 'economic polarization, supply-side economic policies, and the disappearance of the middle class.', 'the decline of colonial powers, the rise of new independent nations, and the spread of democracy.', 'the introduction of new technologies, the rise of the internet, and the growth of multinational corporations.', 'the rise of the civil rights movement, the spread of communism, and the threat of nuclear war.', 'the end of the Cold War, the dissolution of the Soviet Union, and the rise of globalization.', 'rapid industrialization, urban growth and congestion, and corporate consolidation.']\",\"question\":\"This question refers to the following information.\\n\\\"When we were kids the United States was the wealthiest and strongest country in the world; the only one with the atom bomb, the least scarred by modern war, an initiator of the United Nations that we thought would distribute Western influence throughout the world. Freedom and equality for each individual, government of, by, and for the people—these American values we found good, principles by which we could live as men. Many of us began maturing in complacency.\\n\\\"As we grew, however, our comfort was penetrated by events too troubling to dismiss. First, the permeating and victimizing fact of human degradation, symbolized by the Southern struggle against racial bigotry, compelled most of us from silence to activism. Second, the enclosing fact of the Cold War, symbolized by the presence of the Bomb, brought awareness that we ourselves, and our friends, and millions of abstract 'others' we knew more directly because of our common peril, might die at any time. . . .\\\"\\n—Port Huron Statement, 1962\\nThrough the remainder of the 1960s, the growth of the organization that published the Port Huron Statement can best be understood in the context of\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}