{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b9321061-8306-5409-8ffc-03b49057e79d","task_key":"all--test--b9321061-8306-5409-8ffc-03b49057e79d","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"[\\\"the Bolsheviks' need to end the Russian war effort in order to consolidate their revolutionary gains\\\", 'corruption on the part of Bolshevik leaders and collaboration with Russian business interests', 'the breaking up of the Triple Entente', 'French and British aid being given to the so-called White Russians who opposed the Bolshevik government']\",\"question\":\"This question refers to the following information.\\nARTICLE I: Germany, Austria-Hungary, Bulgaria and Turkey for the one part, and Russia for the other part, declare that the state of war between them has ceased. They are resolved to live henceforth in peace and amity with one another. …\\nARTICLE III: The territories lying to the west of the line agreed upon by the contracting parties, which formerly belonged to Russia, will no longer be subject to Russian sovereignty; the line agreed upon is traced on the map submitted as an essential part of this treaty of peace. The exact fixation of the line will be established by a Russo-German commission.\\nTreaty of Brest-Litovsk, March 14, 1918\\nThe Treaty of Brest-Litovsk can best be described as the result of\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}