{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2cc7a59f-2ee9-5545-8576-21b95e56a2f1","task_key":"all--test--2cc7a59f-2ee9-5545-8576-21b95e56a2f1","task_revision_id":"1","upstream_id":"","short_description":"A 33-year-old male physician reports for a shift in the emergency department. A…","config":"all","split":"test","body":"{\"choices\":\"['Ask the physician if he is sober, and if he says yes, allow him to complete his shift', 'Explain to the physician that you suspect he is intoxicated and ask him to submit to a blood sample to check his blood alcohol concentration', \\\"Relieve the physician of duty and alert the hospital's patient safety officer\\\", 'Tell the physician that you can cover the remainder of the shift alone, so that he can sleep in his office']\",\"question\":\"A 33-year-old male physician reports for a shift in the emergency department. A nurse alerts you that he noticed a faint odor of alcohol near the physician. When approached, the physician appears tired and more disheveled than usual. There is an odor of alcohol on his breath. He is a skilled and talented physician with no known history of substance or alcohol use disorder. He is married with three children, and his wife is pregnant with twins. The physician was recently hired and has had no actions on his license by any state medical board. Which of the following is the most appropriate next step?\",\"subject\":\"professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}