{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be401fda-8b13-57ce-b91c-f11dd912a519","task_key":"default--test--8807","task_revision_id":"1","upstream_id":"8807","short_description":"A ladder 9 meters in length is leaning against a vertical wall on level ground.…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['12*sqrt(2)', '6*sqrt(2)', '4*sqrt(2)', 'sqrt(2)', '8*sqrt(2)', '1/(2*sqrt(2))', '5*sqrt(2)', '2*sqrt(2)', '3*sqrt(2)', '10*sqrt(2)']\",\"question\":\"A ladder 9 meters in length is leaning against a vertical wall on level ground. As the bottom end of the ladder is moved away from the wall at a constant rate of 2 meters per second, the top end slides downward along the wall. How fast, in meters per second, will the top end of the ladder be sliding downward at the moment the top end is 3 meters above the ground?\",\"src\":\"ori_mmlu-college_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}