{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4ce679dd-09b2-5b80-840d-944e51329dab","task_key":"default--test--885","task_revision_id":"1","upstream_id":"885","short_description":"Maine's famous aphorism that 'the movement of progressive societies has hitherto…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['His concept of status is misrepresented.', 'It is wrongly considered to be a universal rule.', 'It is misinterpreted as a prediction.', 'His idea is considered inapplicable to Western legal systems.', 'It is incorrectly related to economic progression.', 'It is taken literally.', 'The historical context of the aphorism is often overlooked.', \\\"His use of the word 'progressive' is often misconstrued.\\\", \\\"His concept of 'contract' is often misinterpreted.\\\", 'It is mistakenly thought to apply to all types of societies.']\",\"question\":\"Maine's famous aphorism that 'the movement of progressive societies has hitherto been a movement from Status to Contract' is often misunderstood. In what way?\",\"src\":\"ori_mmlu-jurisprudence\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}