{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"252a6f27-3817-5e47-a08f-105f803dac1f","task_key":"default--test--4995","task_revision_id":"1","upstream_id":"4995","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Indian National Congress gaining political influence', 'The spread of industrialization in India', 'The spread of communism in Asia', 'World War II', 'The partition of India and Pakistan in 1947', 'The aftermath of World War I', 'The rise of the civil rights movement in the United States', 'The decolonization process already underway in Africa', 'The Great Depression that started in 1929', 'Pressure on colonial powers by the Soviet Union to retreat from their colonies']\",\"question\":\"This question refers to the following information.\\n\\\"MAHATMA GANDHI'S MESSAGE.\\nREMAIN NON-VIOLENT\\nThe following message was sent by Mahatma Gandhi from Waltair:—\\nMaulana Mohamed AH was arrested at Waltair under sections 107 and 108 to be called upon to give security, to be of good behaviour for one year. The place and date of trial is unknown.\\nThe Begum Saheba and Mr. Hayat were permitted to see him after arrest.\\nHe and I were going to address a meeting outside the station. He was arrested. I continued going to the meeting and addressed them. There is no cause for sorrow, but every cause for congratulation. There should be no hartal. Perfect peace and calmness should be observed. I regard the arrest as a prelude to Swaraj and the redress of the Khilafat and the Punjab wrongs, if we can remain non-violent. Retain Hindu-Muslim Unity despite the madness of some Moplahs, and fulfil the Swadeshi programme.\\nI hope every Indian, man or woman, will completely boycott foreign cloth and take up spinning or weaving during every spare minute.\\nBy striving like the Maulana, be insistent on religious and national rights.\\nLet us earn imprisonment. I am conscious of the Maulana's innocence and I am sure the imprisonment of the innocent will enable the nation to reach the cherished goal.\\\"\\nMohandas Gandhi, The Pilgrims' March, 1921\\nWhich of the following historical developments most assisted the ultimate success of the boycott mentioned in the fifth paragraph?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}