{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"106b618e-1eae-5cd9-af1c-a61d6c7d445d","task_key":"default--test--4723","task_revision_id":"1","upstream_id":"4723","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['was supported by other European nations after World War II', \\\"was reinforced by Germany's economic recovery after World War II\\\", \\\"was proven correct by Germany's defeat in World War II\\\", \\\"resulted from Germany's defeat in World War II\\\", 'was the prevailing view in Germany after World War II', 'was rejected by all Germans after World War II', \\\"was discredited by Germany's defeat in World War II\\\", \\\"was unaffected by Germany's defeat in World War II\\\", \\\"led to Germany's victory in World War II\\\"]\",\"question\":\"This question refers to the following information.\\nFor a long time, educated Germans answered it in the positive, initially by laying claim to a special German mission, then, after the collapse of 1945, by criticizing Germany's deviation from the West. Today, the negative view is predominant. Germany did not, according to the now prevailing opinion, differ from the great European nations to an extent that would justify speaking of a \\\"unique German path.\\\" And, in any case, no country on earth ever took what can be described as the \\\"normal path.\\\"\\nHeinrich August Winkler, Germany: The Long Road West, Volume 1, 2006\\nFrom this passage, one may infer that the belief that Germany had a unique mission in history\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}