{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"eadcb316-8aee-54d2-952d-e6f9152a0557","task_key":"default--test--4913","task_revision_id":"1","upstream_id":"4913","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the policy of containment.', 'the start of the Cold War.', 'the establishment of the United Nations.', 'participation in the Atlantic Charter.', 'the creation of NATO.', 'embarking on a \\\"roll-back\\\" of communism.', 'the principle of \\\"massive retaliation.\\\"', 'the policy of détente.']\",\"question\":\"This question refers to the following information.\\n\\\"The seeds of totalitarian regimes are nurtured by misery and want. They spread and grow in the evil soil of poverty and strife. They reach their full growth when the hope of a people for a better life has died. We must keep that hope alive. . . . Great responsibilities have been placed upon us by the swift movement of events. . . . I am confident that the Congress will face these responsibilities squarely.\\\"\\n—President Harry S. Truman, 1947\\nThe passage above can best be seen as providing a rationale for\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}