{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d960a34-6da6-51e2-8436-f6fa34d4a487","task_key":"default--test--5205","task_revision_id":"1","upstream_id":"5205","short_description":"Which of the following best describes the fate of most popular third-party…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['They displace one of the two major parties and become major parties themselves.', 'They are ultimately abandoned by the public because their politics are perceived as too radical.', 'Their supporters become frustrated and withdraw from the political process.', \\\"They disintegrate when one or both of the major parties adopt the third party's goals.\\\"]\",\"question\":\"Which of the following best describes the fate of most popular third-party movements?\",\"src\":\"ori_mmlu-high_school_government_and_politics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}