{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b27b6db0-f6a1-5677-aea1-c14450583276","task_key":"default--test--9317","task_revision_id":"1","upstream_id":"9317","short_description":"The highest ocean tides occur when Earth and the Moon are","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['at a 45-degree angle to the Sun', 'both at their furthest point from the Sun', 'both at their closest point to the Sun', 'lined up during winter', 'at any angle to the Sun', 'furthest from each other', 'at right angles to the Sun', 'lined up with the Sun', 'lined up during spring', 'closest to each other']\",\"question\":\"The highest ocean tides occur when Earth and the Moon are\",\"src\":\"ori_mmlu-conceptual_physics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}