{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f0161e95-23b5-50af-ab6d-0ec2c3647af3","task_key":"default--test--5231","task_revision_id":"1","upstream_id":"5231","short_description":"Which of the following actions is most important before removing the blade from…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"[\\\"Checking the mower's tire pressure\\\", 'Checking the oil level', 'Removing the mower bag', 'Removing the spark plug', 'Sharpening the blade before removal', 'Adjusting the wheels to their highest level', 'Cleaning the mower deck', 'Emptying the gas tank', 'Removing the air filter', 'Releasing the brake']\",\"question\":\"Which of the following actions is most important before removing the blade from a gas-powered lawn mower?\",\"src\":\"ori_mmlu-miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}