{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"85e9c65c-57af-548f-8c9f-0af5db09ae0a","task_key":"default--test--4688","task_revision_id":"1","upstream_id":"4688","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"that Germany's skill at dissimulation was responsible for its victory in the war\\\", 'that the reason Germany lost the war was that it was betrayed from within', \\\"that the lyrics from the popular song Deutschland über alles (which eventually became the German national anthem) were evidence of Germany's aggressive attitude\\\", \\\"that Germany's misfortune in the war was due to its lack of skill in dissimulation\\\", 'that the lyrics from the popular song Deutschland über alles (which eventually became the German national anthem) were the reason Germany started the war', 'that Germany provided the ultimate framework for modern warfare', \\\"that Germany's excessive candour led to its downfall in the war\\\"]\",\"question\":\"This question refers to the following information.\\nFor the catastrophe of 1914 the Germans are responsible. … Germany, in this matter, was unfortunate enough to allow herself (in spite of her skill at dissimulation) to be betrayed into an excess of candour by her characteristic tendency to go to extremes. Deutschland über alles. Germany above everything! … There you have the ultimate framework of an old but childish race.\\nGeorges Clemenceau, Grandeur and Misery of Victory, 1930\\nFrom the passage, one may infer that Clemenceau believed\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}