{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f838b1b6-65b9-5f20-a19c-2f93bb0d9903","task_key":"default--test--884","task_revision_id":"1","upstream_id":"884","short_description":"Maine's famous aphorism that 'the movement of progressive societies has hitherto…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['His concept of status is misrepresented.', 'It is misinterpreted as a prediction.', 'His idea is considered inapplicable to Western legal systems.', 'It is incorrectly related to economic progression.', 'It is taken literally.', \\\"His use of the word 'progressive' is often misconstrued.\\\", \\\"His concept of 'contract' is often misinterpreted.\\\"]\",\"question\":\"Maine's famous aphorism that 'the movement of progressive societies has hitherto been a movement from Status to Contract' is often misunderstood. In what way?\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}