{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fe86b5dd-53e3-5f42-893d-1509177b519e","task_key":"all--test--fe86b5dd-53e3-5f42-893d-1509177b519e","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['the industrial revolution in the context of German history', 'the failure of the revolutions of 1848', 'the rise of the Nazi Party in Germany', 'the importance of the revolutions of 1848 in modern German history']\",\"question\":\"This question refers to the following information.\\nRead the the following quotation to answer questions.\\n1848 was the decisive year of German, and so of European, history: it recapitulated Germany's past and inspired Germany's future. … Never has there been a revolution so inspired by a limitless faith in the power of ideas; never has a revolution so discredited the power of ideas in its result. The success of the revolution discredited conservative ideas; the failure of the revolution discredited liberal ideas. After it, nothing remained but the idea of Force, and this idea stood at the helm of German history from then on. For the first time since 1521, the German people stepped on to the centre of the German stage only to miss their cue once more. German history reached its turning-point and failed to turn. This was the fateful essence of 1848.\\nA. J. P. Taylor, The Course of German History, 1945\\nThe subject of Taylor's analysis in this quotation is\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}