{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f630bd8-adba-5d94-aa34-7727ca736bc4","task_key":"default--test--4886","task_revision_id":"1","upstream_id":"4886","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Exterminating all Indians', 'Building reservations and encouraging Indians to change their ways', 'Completely integrating Indians into American society', 'Encouraging Indians to migrate to Canada and Mexico', 'Creating a separate state for Indians', 'Placing Indians in internment camps', 'Respecting Indian territory and sovereignty', 'Providing Indians with financial compensation for land taken by white settlers', 'Ignoring Indian affairs and leaving them to their own devices', 'Encouraging Indians to assimilate into white society']\",\"question\":\"This question refers to the following information.\\nI appeal to any white man to say, if ever he entered Logan's cabin hungry, and he gave him not meat: if ever he came cold and naked, and he clothed him not? During the course of the last long and bloody war, Logan remained idle in his cabin, an advocate for peace. Such was my love for the whites, that my countrymen pointed as they passed, and said, \\\"Logan is the friend of the white man.\\\" I had even thought to have lived with you but for the injuries of one man. Colonel Cresap, the last spring, in cold blood and unprovoked; murdered all the relations of Logan, not even sparing my women and children. There runs not a drop of my blood in the veins of any living creature. This called on me for revenge. I have sought it: I have killed many; I have fully glutted my vengeance. For my country, I rejoice at the beams of peace. But do not harbor a thought that mine is the joy of fear. Logan never felt fear. He will not turn on his heel to save his life. Who is there to mourn for Logan? Not one.\\n—Address attributed to Logan, an Indian leader, 1774\\nA sympathetic reader of Logan's Address in the early years of the United States would be most likely to support which of the following Indian policies?\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}