{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"977a3bd3-8314-54b7-adf3-2ffbaab3d906","task_key":"default--test--10982","task_revision_id":"1","upstream_id":"10982","short_description":"Logical behaviorism is at odds with our commonsense intuition that mental states…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['cause behavior', 'are the same as behavioral states', 'exist', 'can be directly observed', 'are always visible', 'never interact with each other', 'are physical states', 'can be measured quantitatively', 'never cause behavior', 'do not exist']\",\"question\":\"Logical behaviorism is at odds with our commonsense intuition that mental states _____.\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}