{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"782a8084-e76b-528f-aa05-cea744c80f55","task_key":"default--test--5014","task_revision_id":"1","upstream_id":"5014","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The New Deal', 'The Fair Deal', 'The New Frontier', 'Reconstruction', 'The Square Deal', 'The Great Society', 'The Progressive Era', 'The War on Poverty', 'The Emancipation Proclamation', 'The Civil Rights Act']\",\"question\":\"This question refers to the following information.\\n\\\"We conclude that, in the field of public education, the doctrine of \\\"separate but equal\\\" has no place. Separate educational facilities are inherently unequal. Therefore, we hold that the plaintiffs and others similarly situated for whom the actions have been brought are, by reason of the segregation complained of, deprived of the equal protection of the laws guaranteed by the Fourteenth Amendment.\\\"\\nBrown v. Board of Education, 1954\\nDesegregation of schools was, in part, a response to unfulfilled promises from which of the following initiatives?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}