{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"028408e9-e628-5e4b-a8ca-0176ac914850","task_key":"default--test--4800","task_revision_id":"1","upstream_id":"4800","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the surrender of the eastern part of the Austrian Empire to the Russian Empire', 'the surrender of Bulgaria to the Russian Empire', 'the surrender of the western part of the German Empire to the Russian Empire', 'the establishment of a neutral zone between the German and Russian Empires', 'the surrender of Austria-Hungary to the Russian Empire', 'the surrender of the eastern part of the German Empire to the Russian Empire', 'the establishment of a joint Russo-German control over the western part of the Russian Empire', 'the surrender of the eastern part of the German Empire to the Austrian Empire', 'the surrender of the eastern part of the Russian Empire to the German Empire', 'the surrender of the western part of the Russian Empire to the German Empire']\",\"question\":\"This question refers to the following information.\\nARTICLE I: Germany, Austria-Hungary, Bulgaria and Turkey for the one part, and Russia for the other part, declare that the state of war between them has ceased. They are resolved to live henceforth in peace and amity with one another. …\\nARTICLE III: The territories lying to the west of the line agreed upon by the contracting parties, which formerly belonged to Russia, will no longer be subject to Russian sovereignty; the line agreed upon is traced on the map submitted as an essential part of this treaty of peace. The exact fixation of the line will be established by a Russo-German commission.\\nTreaty of Brest-Litovsk, March 14, 1918\\nThe result of Article III of the treaty was\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}