{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"491b2dd1-d22d-5f67-b20a-5eb6c61e6b18","task_key":"default--test--8595","task_revision_id":"1","upstream_id":"8595","short_description":"A novel has 400 pages and Veronica wants to estimate how long it will take her…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['375', '275', '425', '320', '300', '350', '400', '225', '450', '500']\",\"question\":\"A novel has 400 pages and Veronica wants to estimate how long it will take her to complete it. She reads 250 words per minute. She counted the words on one quarter of a typical page and found that there were approximately 200 words per page. To the nearest minute, how long will it take Veronica to read the book?\",\"src\":\"ori_mmlu-high_school_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}