{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5718290a-3d75-5bd5-adfb-30b3803e01c5","task_key":"default--test--4878","task_revision_id":"1","upstream_id":"4878","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Abolishing Social Security and other welfare programs', 'Announcing that he would not run for re-election', 'Launching the Second New Deal, a series of legislative acts including Social Security', 'Launching a military campaign to divert public attention', \\\"Diverting people's attention by allowing the Japanese to bomb Pearl Harbor\\\", 'Proposing the \\\"Every Man a King\\\" policy', 'Implementing a policy to limit individual wealth to $50,000,000', 'Opposing any form of wealth redistribution', 'Proposing a guaranteed minimum income for all families', 'Urging Congress to cut taxes and deregulate the economy']\",\"question\":\"This question refers to the following information.\\nNow, we have organized a society, and we call it \\\"Share Our Wealth Society,\\\" a society with the motto \\\"Every Man a King.\\\"…\\nWe propose to limit the wealth of big men in the country. There is an average of $15,000 in wealth to every family in America. That is right here today.\\nWe do not propose to divide it up equally. We do not propose a division of wealth, but we do propose to limit poverty that we will allow to be inflicted on any man's family. We will not say we are going to try to guarantee any equality … but we do say that one third of the average is low enough for any one family to hold, that there should be a guarantee of a family wealth of around $5,000; enough for a home, an automobile, a radio, and the ordinary conveniences, and the opportunity to educate their children.…\\nWe will have to limit fortunes. Our present plan is that we will allow no man to own more than $50,000,000. We think that with that limit we will be able to carry out the balance of the program.\\n—Senator Huey P. Long of Louisiana, Radio Address, February 23, 1934\\nPresident Franklin D. Roosevelt responded to political challenges like that of Senator Long by doing which of the following?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}