{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6261673b-1c9d-5715-a9c9-1e89baab686e","task_key":"default--test--4947","task_revision_id":"1","upstream_id":"4947","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['mobilization for war in 1914 was greeted with great sorrow in Paris', 'mobilization for war in 1914 was met with fear and apprehension in Paris', 'mobilization for war in 1914 was greeted with great skepticism in Paris', 'mobilization for war in 1914 was met with indifference in Paris', 'mobilization for war in 1914 was greeted with great sorrow all across Europe', 'mobilization for war in 1914 was met with indifference all across Europe', 'mobilization for war in 1914 was greeted with great skepticism all across Europe', 'mobilization for war in 1914 was greeted with great enthusiasm all across Europe', 'mobilization for war in 1914 was met with fear and apprehension all across Europe', 'mobilization for war in 1914 was greeted with great enthusiasm in Paris']\",\"question\":\"This question refers to the following information.\\n\\\"It's come! It's posted at the district mayor's office,\\\" a passerby shouted at me as he ran. I reached the Rue Drout in one leap. … I read the message at a glance. … \\\"The First Day of Mobilization Will Be Sunday, August 2 [1914].\\\" … It was an announcement to a million and a half Frenchmen. … War! … Dead tired but exhilarated, I got back to [my newspaper's office] and burst into the office of Georges Clemenceau, our chief. \\\"What is Paris saying?\\\" he asked me. \\\"It's singing, sir!\\\" \\\"It will be all right then[,\\\" Clemenceau replied].\\nRoland Doregelès, After 50 Years, c. 1965\\nFrom the passage, one may infer that Doregelès recalled that\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}