{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"00d51987-b58f-529d-9367-d4addd94464d","task_key":"default--test--9875","task_revision_id":"1","upstream_id":"9875","short_description":"If you lived on Venus what phases of the Earth would you see? (Assume your…","config":"default","split":"test","body":"{\"category\":\"physics\",\"options\":\"['Waning crescent New Waxing crescent', 'New Waxing crescent First quarter', 'Full Waning crescent New', 'Full moon only', 'First quarter Waxing gibbous Full', 'New only', 'Waning gibbous New Waxing gibbous', 'Waxing gibbous Full Waning gibbous', 'Third quarter Waning crescent New', 'New first quarter full third quarter']\",\"question\":\"If you lived on Venus what phases of the Earth would you see? (Assume your telescope can see through Venus’s thick clouds…)\",\"src\":\"ori_mmlu-astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}