{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cb3cb7f7-cfb9-563f-a847-96de10873766","task_key":"all--test--cb3cb7f7-cfb9-563f-a847-96de10873766","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['drive the French and Portuguese governments into bankruptcy.', 'conquer large parts of Africa in the eighteenth century.', 'establish several colonies along the Atlantic coastline of North America.', 'destroy the Dutch commercial empire.']\",\"question\":\"This question refers to the following information.\\n\\\"I marvel not a little, right worshipful, that since the first discovery of America (which is now full four score and ten years), after so great conquests and plantings of the Spaniards and Portuguese there, that we of England could never have the grace to set fast footing in such fertile and temperate places as are left as yet unpossessed of them. But . . . I conceive great hope that the time approacheth and now is that we of England may share and part stakes [divide the prize] (if we will ourselves) both with the Spaniard and the Portuguese in part of America and other regions as yet undiscovered.\\n\\\"And surely if there were in us that desire to advance the honor of our country which ought to be in every good man, we would not all this while have [neglected] the possessing of these lands which of equity and right appertain unto us, as by the discourses that follow shall appear most plainly.\\\"\\n—Richard Hakluyt, Divers Voyages Touching the Discovery of America and the Islands Adjacent, 1582\\nBy following the ideas of Richard Hakluyt, England was eventually able to\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}