{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fccf885f-d5fd-5d2e-a1f9-e63bd252b6db","task_key":"default--test--8678","task_revision_id":"1","upstream_id":"8678","short_description":"A shelf has four books on it. The weight, in pounds, of each of the four books…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['w > 3.2', 'w > 2.9', 'w > 2.3', 'w > 2.5', 'w < 3.2', 'w < 2.4', 'w < 2.7', 'w > 3.0', 'w < 3.3', 'w < 3.1']\",\"question\":\"A shelf has four books on it. The weight, in pounds, of each of the four books on the shelf is 2.5, 3.2, 2.7, 2.3 Which inequality represents the weight, w, of any book chosen from the shelf?\",\"src\":\"ori_mmlu-elementary_mathematics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}