{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"15c9d127-3556-52fe-87a5-6a0d59e541c3","task_key":"all--test--15c9d127-3556-52fe-87a5-6a0d59e541c3","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['the need for perestroika had come upon Soviet society suddenly', 'the need for perestroika had been exaggerated', 'the need for perestroika was long in the making', 'the time for perestroika had passed']\",\"question\":\"This question refers to the following information.\\nPerestroika [Restructuring] is an urgent necessity arising from the profound processes of development in our socialist society. This society is ripe for change. It has long been yearning for it. Any delay in beginning perestroika could have led to an exacerbated internal situation in the near future, which, to put it bluntly, would have been fraught with serious social, economic, and political crises.\\nMikhail Gorbachev, Perestroika: New Thinking for Our Country and the World, 1987\\nFrom the passage, one may infer that Gorbachev argued that\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}