{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa8425fc-68f4-57b5-83c4-9d33dde24f8f","task_key":"default--test--11204","task_revision_id":"1","upstream_id":"11204","short_description":"Which is one of the most fundamental themes of Japanese culture and…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Karma and the cycle of rebirth govern life and death', 'The universe is inherently chaotic and unpredictable', 'God is eternal and merciful to those who offer worship', 'The world is inherently evil and suffering is inevitable', 'Life is an illusion and the material world is meaningless', 'Justice will be the final outcome in the next world', 'The pursuit of individualism is a core value', 'Power and force are the most divine attributes', 'The survival of the fittest is an essential principle']\",\"question\":\"Which is one of the most fundamental themes of Japanese culture and civilization?\",\"src\":\"ori_mmlu-world_religions\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}