{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0778d8ff-3c73-55f8-a475-5f77a2366f69","task_key":"default--test--4953","task_revision_id":"1","upstream_id":"4953","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Political corruption', 'Civil disobedience', 'Rules of royal succession', 'Gender inequality', 'Social class conflicts', 'Diplomatic relations between nations', 'Environmental concerns', 'Religious conflict', 'Economic instability', 'Military stalemate']\",\"question\":\"This question refers to the following information.\\nOh, shame on you, poor Winter King!\\nWhat's this that you have done?\\nIs 't not a very naughty thing\\nTo snatch the kaiser's crown?\\nNow you will have to stay away\\nAlike from Rhine and Prague,\\nAnd more than that-shame and dismay\\nYour days and nights will plague.\\nRight well you knew, and all the world,\\nRight well they know this thing,\\nThat Ferdinand alone can be\\nBohemia's lawful king.\\nSo come, dear Fritz, rouse up and go\\nTo Ferdinand, your king,\\nAnd beg him graciously to show\\nFull pardon for your sin.\\n\\\"Poor Winter King,\\\" seventeenth-century song\\nWhich of the following is an important underlying issue in this song?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}