{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64c860b2-9bfd-5574-9804-4578337161a2","task_key":"default--test--6399","task_revision_id":"1","upstream_id":"6399","short_description":"What is a tracheostomy?","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['An opening in the anterior chest wall.', 'An opening in the anterior wall of the trachea below the cricoid cartilage.', 'An opening in the anterior chest wall, above the diaphragm.', 'An opening in the posterior wall of the esophagus.', 'An opening in the posterior wall of the trachea above the cricoid cartilage.', 'An opening in the posterior wall of the trachea below the cricoid cartilage.', 'An opening in the lateral wall of the trachea below the cricoid cartilage.', 'An opening in the lateral wall of the trachea above the cricoid cartilage.', 'An opening in the anterior wall of the trachea above the cricoid cartilage.', 'An opening in the anterior neck, below the thyroid cartilage.']\",\"question\":\"What is a tracheostomy?\",\"src\":\"ori_mmlu-clinical_knowledge\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}