{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"71315911-ecfe-569e-80bb-f3a5849bfe55","task_key":"default--test--10795","task_revision_id":"1","upstream_id":"10795","short_description":"According to Jaina traditions, who were the caityavasis?","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Monks who lived in forests', 'Jain kings who patronized the faith', 'Orthodox interpreters of the scriptures', 'Monks who performed public acts of penance', 'Women ascetics who lived in communities', 'Lay followers who lived in cities', 'Temple-dwelling renouncers', 'Ascetics who lived in isolation', 'Householders who supported renouncers', 'Practitioners of the highest form of asceticism']\",\"question\":\" According to Jaina traditions, who were the caityavasis?\",\"src\":\"ori_mmlu-world_religions\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}