{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"263c27f3-184d-51e2-be27-309ba1fe8698","task_key":"default--test--6447","task_revision_id":"1","upstream_id":"6447","short_description":"A 19-year-old woman comes to the emergency department because, she says, \"I'm…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Increased intensity of the murmur with forced expiration', 'Decreased intensity of the murmur with forced expiration', 'No change in the intensity of the murmur with breathing changes', 'Negative Kussmaul sign (no rise in jugular venous pulse with inspiration)', 'Increased intensity of the murmur with deep inspiration', 'Decreased intensity of S1', 'Positive Kussmaul sign (rise in jugular venous pulse with inspiration)', 'Increased intensity of S1', 'Positive JVP (rise in jugular venous pulse) with expiration', 'Decreased intensity of the murmur with deep inspiration']\",\"question\":\"A 19-year-old woman comes to the emergency department because, she says, \\\"I'm burning up.\\\" Medical history is significant for intravenous drug use. Physical examination discloses a systolic heart murmur over the precordium. An expected physical finding will be which of the following? \",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}