{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4c674cf1-6e7d-524b-8070-5860c2747bb7","task_key":"default--test--4776","task_revision_id":"1","upstream_id":"4776","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Claimed the lands of Bohemia for the Catholic Church', 'Allied with King Ferdinand against other kingdoms', 'Stole the crown from the kaiser', 'Was granted full pardon for his actions by King Ferdinand', 'Was a symbolic figure without real power', 'Was a popular folk hero celebrated in songs and stories', 'Successfully defended his land from the Holy Roman Empire', 'Rebelled against King Ferdinand', 'Gave over his crown to King Ferdinand']\",\"question\":\"This question refers to the following information.\\nOh, shame on you, poor Winter King!\\nWhat's this that you have done?\\nIs 't not a very naughty thing\\nTo snatch the kaiser's crown?\\nNow you will have to stay away\\nAlike from Rhine and Prague,\\nAnd more than that-shame and dismay\\nYour days and nights will plague.\\nRight well you knew, and all the world,\\nRight well they know this thing,\\nThat Ferdinand alone can be\\nBohemia's lawful king.\\nSo come, dear Fritz, rouse up and go\\nTo Ferdinand, your king,\\nAnd beg him graciously to show\\nFull pardon for your sin.\\n\\\"Poor Winter King,\\\" seventeenth-century song\\nBased on the song, the Winter King did which of the following?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}