{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"120669cf-ba6e-5d9d-927f-b269b84c7682","task_key":"default--test--2164","task_revision_id":"1","upstream_id":"2164","short_description":"Kohler's studies with apes are seen as demonstrating","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['that apes are able to use objects from their natural environment as tools.', 'that apes can learn through operant conditioning.', 'that learning can occur via insight.', \\\"the apes' ability to communicate.\\\", 'latent learning in primates.', \\\"the apes' ability to recognize themselves in a mirror.\\\", 'that apes are capable of abstract thinking.', \\\"the apes' ability to mimic human behavior.\\\", \\\"the apes' ability to solve complex mathematical problems.\\\", 'that apes have a basic understanding of human languages.']\",\"question\":\"Kohler's studies with apes are seen as demonstrating\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}