{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"80fc003a-67d8-5708-b893-295a1c35fc85","task_key":"default--test--6357","task_revision_id":"1","upstream_id":"6357","short_description":"A 37-year-old woman with right lower extremity edema is evaluated because of the…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Blue lips or skin', 'Peripheral edema', 'Rapid heart rate', 'Constant chest pain', 'Presence of an S3', 'Wheezing or whistling sound when breathing', 'Shortness of breath after slight activity', 'P2 louder than A2', 'Increased jugular venous pressure', 'Coughing up blood']\",\"question\":\"A 37-year-old woman with right lower extremity edema is evaluated because of the sudden onset of shortness of breath and pleuritic chest pain. A diagnosis of pulmonary embolism is made. Which of the following signs, if present on physical examination, would be the most specific indicator of pulmonary arterial hypertension in this patient?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}