{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54e07cec-6d07-5e6e-a7bd-c596679d8ec7","task_key":"default--test--4974","task_revision_id":"1","upstream_id":"4974","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Banning the slave trade to promote equality among all races.', 'Implementing a system of federal taxation to boost the national economy.', 'Encouraging industrialization and urbanization to improve living standards.', 'Implementing a national education system to improve literacy rates.', 'forging alliances with American Indian nations to present a united front to European powers.', \\\"Establishing a national bank to control the country's finances.\\\", 'Introducing a bill of rights to protect individual liberties.', 'increasing spending on military forces and cutting spending on social programs.', 'adopting a new constitution in order to create a more national government.', 'abandoning an isolationist approach to foreign policy and adopting a more aggressive and interventionist stance.']\",\"question\":\"This question refers to the following information.\\n\\\"Wherever I go—the street, the shop, the house, or the steamboat—I hear the people talk in such a way as to indicate that they are yet unable to conceive of the Negro as possessing any rights at all. Men who are honorable in their dealings with their white neighbors will cheat a Negro without feeling a single twinge of their honor. To kill a Negro they do not deem murder; to debauch a Negro woman they do not think fornication; to take the property away from a Negro they do not consider robbery. The people boast that when they get freedmen affairs in their own hands, to use their own classic expression, 'the niggers will catch hell.'\\n\\\"The reason of all this is simple and manifest. The whites esteem the blacks their property by natural right, and however much they may admit that the individual relations of masters and slaves have been destroyed by the war and the President's emancipation proclamation, they still have an ingrained feeling that the blacks at large belong to the whites at large, and whenever opportunity serves they treat the colored people just as their profit, caprice or passion may dictate.\\\"\\n—Congressional testimony of Col. Samuel Thomas, Assistant Commissioner, Bureau of Refugees, Freedmen and Abandoned Lands, 1865\\nTo address the problems identified in Federalist #15, Hamilton proposed\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}