{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62e4058a-e099-5bee-a86d-2a1129fa67b1","task_key":"default--test--4729","task_revision_id":"1","upstream_id":"4729","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"[\\\"The Bhagavad Gita states that one's duty (dharma) is determined by their caste.\\\", 'The Brahmins were given the highest status because they were priests.', 'Ancient texts describe the Sudras as servants to the other castes.', 'The Rig Veda depicts the caste system as a divine order.', 'The Ramayana, an ancient Indian epic, was written by a prince.', 'For there is nothing better for a Kshatriya than a righteous battle.', 'The person traditionally credited with composing the Mahabarata was born to a fisherwoman.', 'The four divisions of human society are created by me [Krishna].', 'The Hindu God Shiva is considered to be simultaneously the creator, the preserver, and the destroyer.', 'The caste system was religiously sanctioned and had divine origin according to the Manusmriti.']\",\"question\":\"This question refers to the following information.\\n\\\"Purusa-Sukta\\\"\\nPurusa is the lord of the immortals, who grow by means of [ritual] food. When the gods performed a sacrifice with the offering Purusa, spring was its clarified butter, summer the kindling, autumn the oblation.\\nIt was Purusa, born in the beginning, which they sprinkled on the sacred grass as a sacrifice. It made the beasts of the air, the forest and the village. From that sacrifice completely offered, the mantras [Rig Veda] and the songs [Samaveda] were born. The sacrificial formulae [Yajurveda] were born from it. From it the horses were born and all that have cutting teeth in both jaws. The cows were born from it, also. From it were born goats and sheep.\\nWhen they divided Purusa, how many ways did they apportion him? What was his mouth? What were his arms? What were his thighs, his feet declared to be? His mouth was the Brahman, his arms were the Rajanaya [Ksatriya], his thighs the Vaisya; from his feet the Sudra was born. Thus, they fashioned the worlds. The gods sacrificed with the sacrifice to the sacrifice. These were the first rites.\\n—Hymns excerpted and adapted from the Rig Veda, oldest surviving literary work, India, 1500–1000 BCE\\nWhich of the following would best contradict the argument that the caste system imposed rigid economic and social roles on Indian society in the period 600 BCE to 600 CE?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}