{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7dc0d568-faf7-537c-8d34-e37ee9fd6178","task_key":"default--test--4797","task_revision_id":"1","upstream_id":"4797","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Universal female suffrage', 'Equal distribution of wealth', 'Universal male suffrage', 'Universal suffrage', 'Universal health care', 'Universal education', 'Abolition of child labor', 'Establishment of a socialist state', 'Abolition of monarchy', 'Social security and unemployment insurance']\",\"question\":\"This question refers to the following information.\\nUnder the name of Chartist well-meaning inconsiderate men and other misled men have in very many cases, all over the country from the extreme west to the extreme east and from Brighton in the south to nearly the extreme north of Scotland, denounced every man who is not a working man, applied to him, the grossest epithets and most atrocious intentions and conduct, have threatened them with vengeance and in some places, have proposed plans for the seizure and division of their property—numbers of misled men and others of bad character, under the self-denomination of Chartists have gone from place to place and in the most violent manner disturbed and dispersed meetings of various kinds. Your Committee object to the words Household Suffrage since under any honest definition of the words—they would exclude a large majority of the men of these kingdoms—and because they have become reasonably obnoxious to the political portion of the working people.\\n—Excerpt of a letter from British social reformer Francis Place, written in 1842\\nThe primary goal of the Chartist movement was which of the following?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}