{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0380e5a8-42ab-5ec4-8e01-d224514a9615","task_key":"default--test--6356","task_revision_id":"1","upstream_id":"6356","short_description":"A 46-year-old man with Marfan syndrome, aortic insufficiency, and mitral…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Dissection of the aorta', 'Pulmonary embolism', 'Mitral valve prolapse', 'Gastroesophageal reflux disease', 'Chronic obstructive pulmonary disease', 'Ventricular septal defect', 'Stable angina', 'Acute myocardial infarction', 'Esophageal reflux with spasm', 'Acute bacterial endocarditis']\",\"question\":\"A 46-year-old man with Marfan syndrome, aortic insufficiency, and mitral regurgitation comes to the emergency department because he has had severe substernal chest pain for the past 3 hours. He describes the pain as tearing in quality and radiating to the neck. One week earlier he experienced similar but less severe chest pain and treated himself with aspirin. Which of the following is the most likely underlying cause for his worsening symptoms?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}