{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7315a9ae-5cde-5f69-88ed-47dc200b8122","task_key":"default--test--4828","task_revision_id":"1","upstream_id":"4828","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['had already started making strategies for the upcoming war', \\\"was in disagreement with the general public's reaction to the war\\\", 'was unaware of the announcement of war until informed by the narrator', \\\"had grave doubts about France's ability to win a war against Germany\\\", 'was fearful of the upcoming war and its potential consequences', 'was relieved to hear that Paris was reacting with joy at the advent of war', 'would one day serve as prime minister of France', 'believed that the war would bring economic prosperity to France', 'was indifferent to the reaction of Paris towards the war', 'was disgusted to hear that Paris was reacting with joy at the advent of war']\",\"question\":\"This question refers to the following information.\\n\\\"It's come! It's posted at the district mayor's office,\\\" a passerby shouted at me as he ran. I reached the Rue Drout in one leap. … I read the message at a glance. … \\\"The First Day of Mobilization Will Be Sunday, August 2 [1914].\\\" … It was an announcement to a million and a half Frenchmen. … War! … Dead tired but exhilarated, I got back to [my newspaper's office] and burst into the office of Georges Clemenceau, our chief. \\\"What is Paris saying?\\\" he asked me. \\\"It's singing, sir!\\\" \\\"It will be all right then[,\\\" Clemenceau replied].\\nRoland Doregelès, After 50 Years, c. 1965\\nFrom the passage, one may infer that Clemenceau\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}