{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"05005354-462d-5004-b598-bcc8e8606724","task_key":"default--test--11015","task_revision_id":"1","upstream_id":"11015","short_description":"Philosophy is concerned primarily with identifying beliefs about human existence…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"who really believes X? and how can we explain differences in people's beliefs?\\\", 'what is the origin of thought? and how does it impact our actions?', 'what do you mean? and how do you know?', 'how do beliefs shape society? and why do beliefs differ?', 'how do philosophers argue? and are their differences important?', 'why is there something rather than nothing? and how can we be sure?', \\\"how can we prove it? and what if we're wrong?\\\", 'why should we bother? and what are the consequences of our believing one thing over another?']\",\"question\":\"Philosophy is concerned primarily with identifying beliefs about human existence and evaluating arguments that support those beliefs.  These activities can be summarized in two questions that drive philosophical investigations:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}