{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a32576c3-05fa-557c-b0ee-d03b4c09b704","task_key":"all--test--a32576c3-05fa-557c-b0ee-d03b4c09b704","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['cultural and intellectual arguments justified the success of those at the top of the socioeconomic structure as both appropriate and inevitable.', 'popular writers rejected ideas from the sciences, and based their arguments on faith.', 'intellectuals were critical of the cut-throat competition of the ages, and proposed radical alternatives based on creating a cooperative economy.', 'cultural products of the era tended to ignore the economic direction of society and looked back wistfully to the past.']\",\"question\":\"This question refers to the following information.\\n\\\"A drunkard in the gutter is just where he ought to be. . . . The law of survival of the fittest was not made by man, and it cannot be abrogated by man. We can only, by interfering with it, produce the survival of the unfittest. . . . The millionaires are a product of natural selection, acting on the whole body of men to pick out those who can meet the requirement of certain work to be done. In this respect they are just like the great statesmen, or scientific men, or military men. It is because they are thus selected that wealth—both their own and that entrusted to them—aggregates under their hands. Let one of them make a mistake and see how quickly the concentration gives way to dispersion.\\\"\\n—William Graham Sumner, What Social Classes Owe to Each Other, 1883\\nThe sociological ideas of William Graham Sumner reflect the idea that during the late 1800s\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}