{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f700ac9f-e036-5eb6-a2a7-3f10ec5b3686","task_key":"default--test--4957","task_revision_id":"1","upstream_id":"4957","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Roman Catholic', 'Muslim', 'Agnostic', 'Hindu', 'Atheist', 'Eastern Orthodox', 'Buddhist', 'Sikh', 'Protestant', 'Jewish']\",\"question\":\"This question refers to the following information.\\nOh, shame on you, poor Winter King!\\nWhat's this that you have done?\\nIs 't not a very naughty thing\\nTo snatch the kaiser's crown?\\nNow you will have to stay away\\nAlike from Rhine and Prague,\\nAnd more than that-shame and dismay\\nYour days and nights will plague.\\nRight well you knew, and all the world,\\nRight well they know this thing,\\nThat Ferdinand alone can be\\nBohemia's lawful king.\\nSo come, dear Fritz, rouse up and go\\nTo Ferdinand, your king,\\nAnd beg him graciously to show\\nFull pardon for your sin.\\n\\\"Poor Winter King,\\\" seventeenth-century song\\nThe narrator of the song would most likely identify as which of the following?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}