{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"24af59b2-3998-54e2-994f-4c1be1e323de","task_key":"default--test--2523","task_revision_id":"1","upstream_id":"2523","short_description":"A key accomplishment of Piaget's preoperational stage is the emergence of:","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['the understanding of cause and effect.', 'the capacity for deductive reasoning.', 'the ability to reason about abstract concepts and problems.', 'the symbolic function.', 'the object concept.', 'abstract reasoning.', \\\"the ability to understand others' perspectives.\\\", 'formal operations.', 'the ability to handle multiple variables in systematic ways.', 'conservation.']\",\"question\":\"A key accomplishment of Piaget's preoperational stage is the emergence of:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}