{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8c91edf7-a6d0-5ad3-9a9e-86559b98561a","task_key":"default--test--6446","task_revision_id":"1","upstream_id":"6446","short_description":"A 19-year-old woman comes to the emergency department because, she says, \"I'm…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Decreased intensity of the murmur with deep inspiration', 'Increased intensity of the murmur with forced expiration', 'Increased intensity of S1', 'Decreased intensity of the murmur with forced expiration', 'Positive JVP (rise in jugular venous pulse) with expiration', 'No change in the intensity of the murmur with breathing changes', 'Positive Kussmaul sign (rise in jugular venous pulse with inspiration)', 'Increased intensity of the murmur with deep inspiration', 'Negative Kussmaul sign (no rise in jugular venous pulse with inspiration)', 'Decreased intensity of S1']\",\"question\":\"A 19-year-old woman comes to the emergency department because, she says, \\\"I'm burning up.\\\" Medical history is significant for intravenous drug use. Physical examination discloses a systolic heart murmur over the precordium. An expected physical finding will be which of the following?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}