{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"68c5ef2c-180c-51dd-b2f0-d9d2a2aebde5","task_key":"default--test--6358","task_revision_id":"1","upstream_id":"6358","short_description":"A 37-year-old woman with right lower extremity edema is evaluated because of the…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Wheezing or whistling sound when breathing', 'Constant chest pain', 'Blue lips or skin', 'Shortness of breath after slight activity', 'P2 louder than A2', 'Peripheral edema', 'Presence of an S3', 'Increased jugular venous pressure', 'Rapid heart rate', 'Coughing up blood']\",\"question\":\"A 37-year-old woman with right lower extremity edema is evaluated because of the sudden onset of shortness of breath and pleuritic chest pain. A diagnosis of pulmonary embolism is made. Which of the following signs, if present on physical examination, would be the most specific indicator of pulmonary arterial hypertension in this patient? \",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}