{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b2cfd8d7-2317-567a-a0c5-c694fdb22c9e","task_key":"default--test--6239","task_revision_id":"1","upstream_id":"6239","short_description":"A 24-year-old recent college graduate comes to the office because of a 3-year…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Recommend a diet change', 'Suggest a neurological examination', 'Recommend behavioral therapy', \\\"Review the patient's sexual history\\\", 'Recommend that the patient avoid any stressful activities that cause the problem', 'Recommend a urinary incontinence medication', 'Recommend a physical therapy', 'Advise the patient to stop taking oral contraceptives', 'Recommend psychoanalytic psychotherapy']\",\"question\":\"A 24-year-old recent college graduate comes to the office because of a 3-year history of increased frequency of urination and occasional incontinence whenever she travels. She says the urinary symptoms typically begin a few days before she is going to fly and they stop the day after she arrives at her destination. She says she is anxious about flying and even slightly \\\"panicky.\\\" She reports having had similar but milder symptoms prior to examinations before graduating. Medical history is otherwise unremarkable. Her only medication is an oral contraceptive. Vital signs are normal. Physical examination discloses no abnormalities. Urinalysis and urine culture are normal. Which of the following is the most appropriate next step?\",\"src\":\"ori_mmlu-professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}