{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"40553c15-5804-58b2-8f97-22ccaf7814ad","task_key":"all--test--40553c15-5804-58b2-8f97-22ccaf7814ad","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Gave over his crown to King Ferdinand', 'Successfully defended his land from the Holy Roman Empire', 'Rebelled against King Ferdinand', 'Claimed the lands of Bohemia for the Catholic Church']\",\"question\":\"This question refers to the following information.\\nOh, shame on you, poor Winter King!\\nWhat's this that you have done?\\nIs 't not a very naughty thing\\nTo snatch the kaiser's crown?\\nNow you will have to stay away\\nAlike from Rhine and Prague,\\nAnd more than that-shame and dismay\\nYour days and nights will plague.\\nRight well you knew, and all the world,\\nRight well they know this thing,\\nThat Ferdinand alone can be\\nBohemia's lawful king.\\nSo come, dear Fritz, rouse up and go\\nTo Ferdinand, your king,\\nAnd beg him graciously to show\\nFull pardon for your sin.\\n\\\"Poor Winter King,\\\" seventeenth-century song\\nBased on the song, the Winter King did which of the following?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}