{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed3551ab-3450-5683-b800-b94902807dc9","task_key":"all--test--ed3551ab-3450-5683-b800-b94902807dc9","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['the surrender of the western part of the German Empire to the Russian Empire', 'the surrender of the eastern part of the German Empire to the Russian Empire', 'the surrender of the western part of the Russian Empire to the German Empire', 'the surrender of the eastern part of the Russian Empire to the German Empire']\",\"question\":\"This question refers to the following information.\\nARTICLE I: Germany, Austria-Hungary, Bulgaria and Turkey for the one part, and Russia for the other part, declare that the state of war between them has ceased. They are resolved to live henceforth in peace and amity with one another. …\\nARTICLE III: The territories lying to the west of the line agreed upon by the contracting parties, which formerly belonged to Russia, will no longer be subject to Russian sovereignty; the line agreed upon is traced on the map submitted as an essential part of this treaty of peace. The exact fixation of the line will be established by a Russo-German commission.\\nTreaty of Brest-Litovsk, March 14, 1918\\nThe result of Article III of the treaty was\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}