{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a3273e7e-5370-5e15-abe7-97a8a3dfb948","task_key":"default--test--4767","task_revision_id":"1","upstream_id":"4767","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Enactment of laws regulating railroads.', 'Institution of the Federal Reserve System.', 'Creation of the National Park Service.', 'Creation of a price-support system for small-scale farmers.', 'Establishment of the social security system.', 'Abolition of child labor.', 'Establishment of the minimum wage law.', 'Introduction of the direct election of U.S. Senators.', 'Passage of the 19th Amendment, granting women the right to vote.', 'Shift in U.S. currency from the gold standard to the silver standard.']\",\"question\":\"This question refers to the following information.\\n\\\"I was once a tool of oppression\\nAnd as green as a sucker could be\\nAnd monopolies banded together\\nTo beat a poor hayseed like me.\\n\\\"The railroads and old party bosses\\nTogether did sweetly agree;\\nAnd they thought there would be little trouble\\nIn working a hayseed like me. . . .\\\"\\n—\\\"The Hayseed\\\"\\nWhich of the following is an accomplishment of the political movement that was organized around sentiments similar to the one in the song lyrics above?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}