{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"395d2b42-a901-5a98-a875-5d6ed6dde366","task_key":"default--test--4802","task_revision_id":"1","upstream_id":"4802","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"Revolutions are always successful in improving women's status and rights.\\\", 'Women rarely support revolutions and therefore receive little benefit from them.', 'Women are typically the primary beneficiaries of revolutionary movements.', \\\"The success of a revolution is determined by its impact on women's rights.\\\", 'The gains of a revolution are usually equally distributed among men and women.', 'Revolutions usually result in immediate gender equality.', 'Revolutionary progressivism does not always concern itself with gender equality.', \\\"Women's rights are always championed by revolutionary leaders.\\\", 'Women typically assume leadership roles during times of revolution.']\",\"question\":\"This question refers to the following information.\\nWoman, wake up; the bell of reason is being heard throughout the whole universe; discover your rights. Enslaved man has multiplied his strength, [but] having become free, he has become unjust to his companion. Oh, women, women! When will you cease to be blind? What advantage have you received from the Revolution? A more pronounced scorn, a more marked disdain. If our leaders persist, courageously oppose the force of reason to their empty pretentions of superiority. Regardless of what barriers confront you, it is in your power to free yourselves!\\nOlympe de Gouges, \\\"Declaration of the Rights of Woman and the Female Citizen,\\\" 1791\\nThe independence? Nothing of what I hoped for was achieved. I had expected that my children would be able to have an education, but they did not get it. We were poor peasants then, we are poor peasants now. Nothing has changed. Everything is the same. The only thing is that we are free, the war is over, we work without fear—but apart from that, nothing has changed.\\nHalima Ghomri, interviewed in the 1970s, after Algeria's war of independence\\nWhich of the following observations about revolutions and gender is best supported by the first passage?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}