{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dabcb7c7-a310-5079-b106-f4077768f05a","task_key":"default--test--4744","task_revision_id":"1","upstream_id":"4744","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"the Servicemen's Readjustment Act (G.I. Bill).\\\", 'an extension of aid to Greece and Turkey.', 'desegregation of the U.S. armed forces', 'the Taft-Hartley Act', 'the McCarran Internal Security Act.', 'the Marshall Plan for European recovery', 'development of the hydrogen bomb.', 'the implementation of the Fair Deal', 'the creation of the United Nations', 'establishment of NATO']\",\"question\":\"This question refers to the following information.\\n\\\"The seeds of totalitarian regimes are nurtured by misery and want. They spread and grow in the evil soil of poverty and strife. They reach their full growth when the hope of a people for a better life has died. We must keep that hope alive. . . . Great responsibilities have been placed upon us by the swift movement of events. . . . I am confident that the Congress will face these responsibilities squarely.\\\"\\n—President Harry S. Truman, 1947\\nThe passage above is part of President Truman's argument to Congress in favor of\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}