{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bf782f66-6cff-544e-8c27-f279a27feb3d","task_key":"default--test--4783","task_revision_id":"1","upstream_id":"4783","short_description":"What do Homo sapiens and Australopithecus afarensis have in common?","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['a shared ability to use complex tools', 'a similar diet', 'a reliance on brachiation', 'an identical social structure', 'a mode of locomotion', 'the same lifespan', 'the same geographical distribution', 'a lack of sexual dimorphism', 'the same language capability', 'all the above']\",\"question\":\"What do Homo sapiens and Australopithecus afarensis have in common?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}