{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66f1d0ff-7735-56ec-825e-e0bebd605192","task_key":"all--test--66f1d0ff-7735-56ec-825e-e0bebd605192","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Peace of Augsburg', 'War of Spanish Succession', 'Edict of Nantes', \\\"Thirty Years' War\\\"]\",\"question\":\"This question refers to the following information.\\nOh, shame on you, poor Winter King!\\nWhat's this that you have done?\\nIs 't not a very naughty thing\\nTo snatch the kaiser's crown?\\nNow you will have to stay away\\nAlike from Rhine and Prague,\\nAnd more than that-shame and dismay\\nYour days and nights will plague.\\nRight well you knew, and all the world,\\nRight well they know this thing,\\nThat Ferdinand alone can be\\nBohemia's lawful king.\\nSo come, dear Fritz, rouse up and go\\nTo Ferdinand, your king,\\nAnd beg him graciously to show\\nFull pardon for your sin.\\n\\\"Poor Winter King,\\\" seventeenth-century song\\nWhich of the following events inspired the composition of this song?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}