{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64d48e51-236f-573f-ac1e-81d9cedb861d","task_key":"all--test--64d48e51-236f-573f-ac1e-81d9cedb861d","task_revision_id":"1","upstream_id":"","short_description":"A 6-month-old male Hispanic infant is brought to the office by his parents…","config":"all","split":"test","body":"{\"choices\":\"['Herniorrhaphy can be postponed until age 2 years because many hernias close spontaneously', 'Herniorrhaphy can be postponed until age 12 years because oligospermia does not develop before age 12', 'Herniorrhaphy should be scheduled at the earliest convenient time', 'Herniorrhaphy should be scheduled as an emergency operation']\",\"question\":\"A 6-month-old male Hispanic infant is brought to the office by his parents because of intermittent swelling of his right scrotum that is more pronounced when he cries. The swelling has never been red or \\\"stuck.\\\" Vital signs are normal. A right inguinal hernia is confirmed on physical examination. In discussing repair of the hernia with the parents, it is most appropriate to inform them of which of the following?\",\"subject\":\"professional_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}