{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"424ce6d1-6be6-5da4-a709-11641dfb35c9","task_key":"default--test--4848","task_revision_id":"1","upstream_id":"4848","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"the candidates' lack of military service records\\\", 'the rise of third-party candidates that split the Democratic vote', \\\"the American public's desire to avoid conflict and return to a more conservative political and social life\\\", \\\"the Democratic Party's perceived weakness on national security issues\\\", \\\"the Democratic candidates' controversial positions on civil rights legislation\\\", \\\"the Democratic candidates' inability to effectively communicate their policies to the public\\\", \\\"the Democratic Party's unequivocal support of the Equal Rights Amendment\\\", 'the Democratic Party platform pledge to increase taxes in order to pay off the national debt']\",\"question\":\"This question refers to the following information.\\n\\\"We found that not only was it a civil war, an effort by a people who had for years been seeking their liberation from any colonial influence whatsoever, but also we found that the Vietnamese whom we had enthusiastically molded after our own image were hard put to take up the fight against the threat we were supposedly saving them from.\\n\\\"We found most people didn't even know the difference between communism and democracy. They only wanted to work in rice paddies without helicopters strafing them and bombs with napalm burning their villages and tearing their country apart. They wanted everything to do with the war, particularly with this foreign presence of the United States of America, to leave them alone in peace, and they practiced the art of survival by siding with whichever military force was present at a particular time, be it Viet Cong, North Vietnamese or American.\\\"\\nJohn Kerry, 1971\\nThe most important factor in the defeat of Democratic presidential candidates in the elections of 1952 and 1968 was\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}