{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4a3057c4-46ee-56ba-97f1-1955f81fb77d","task_key":"default--test--5751","task_revision_id":"1","upstream_id":"5751","short_description":"Consider the following statements: (1) In every country, people think other…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Both statements are largely true', 'False, False', 'True, True', 'The first statement is true, the second statement is largely false', 'True, False', 'The first statement is false, the second statement is partially true', 'False, True', 'The first statement is partially true, the second statement is false', 'The first statement is largely false, the second statement is true', 'Both statements are partially true']\",\"question\":\"Consider the following statements: (1) In every country, people think other individuals in their country are less happy than they actually are. (2) In the United States, women’s happiness has increased both absolutely and relative to men. Which of these statements is True/False?\",\"src\":\"ori_mmlu-global_facts\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}