{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"af53b4bf-d72f-50e4-b513-3edb0a6ff62d","task_key":"default--test--7728","task_revision_id":"1","upstream_id":"7728","short_description":"One of the events at the circus was Gabriella, the Human Cannonball. On Saturday…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['60.18 m', '70.18 m', '62.18 m', '57.14 m', '74.18 m', '66.18 m', '72.18 m', '64.18 m', '59.24 m', '68.18 m']\",\"question\":\"One of the events at the circus was Gabriella, the Human Cannonball. On Saturday she did four shows. Her distances measured 7.04 meters, 18.81 meters, 18.39 meters, and 19.94 meters. What was the total distance Gabriella flew that day?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}