{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"155ee2b8-0f5c-5828-acb5-742e060b2cea","task_key":"default--test--4861","task_revision_id":"1","upstream_id":"4861","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The religion is focused on material wealth and success.', 'It believes in multiple universes.', 'It is a polytheistic religion.', 'It believes in reincarnation into various animal forms.', 'It emphasizes on the importance of community over individual faith.', 'There is an expectation of an afterlife.', 'Salvation is based on the correct completion of rituals.', 'The religion rejects the concept of a universal spirit.', 'The religion promotes atheism.']\",\"question\":\"This question refers to the following information.\\nHe contains all works and desires and all perfumes and all tastes. He enfolds the whole universe and in silence is loving to all. This is the Spirit that is in my heart, this is Brahman. To him I shall come when I go beyond this life, and to him will come he who has faith and doubts not.\\n—The Upanishads, India, c. 1000 BCE\\nBased on the quotation, which statement is true of the speaker's religion?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}