{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7fcd548-09d2-5587-8eaf-10850e138183","task_key":"default--test--8860","task_revision_id":"1","upstream_id":"8860","short_description":"A woman walks due west on the deck of a ship at $3 \\mathrm{mi} / \\mathrm{h}$.…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['$\\\\\\\\sqrt{25}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{441}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{475}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{500}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{485}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '3 $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{493}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{529}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '22 $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$', '$\\\\\\\\sqrt{517}$ $\\\\\\\\mathrm{mi}/\\\\\\\\mathrm{h}$']\",\"question\":\"A woman walks due west on the deck of a ship at $3 \\\\mathrm{mi} / \\\\mathrm{h}$. The ship is moving north at a speed of $22 \\\\mathrm{mi} / \\\\mathrm{h}$. Find the speed of the woman relative to the surface of the water.\",\"src\":\"scibench-calculus\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}