{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ad8112fc-5ab0-5985-8be9-dee4e355b072","task_key":"all--test--ad8112fc-5ab0-5985-8be9-dee4e355b072","task_revision_id":"1","upstream_id":"","short_description":"A 3-week-old baby, came down with a very high fever. The baby's pediatrician…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Whether medical treatment is necessary to save the baby's life.\\\", \\\"Whether the mother's refusal to authorize medical treatment is justified on the basis of current knowledge.\\\", 'Whether the hospital is owned and operated by the state.', 'Whether the mother is a taxpayer of the state in which the court is located.']\",\"question\":\"A 3-week-old baby, came down with a very high fever. The baby's pediatrician informed the mother that unless her son received immediate medical treatment, he would die. The mother objected on religious grounds. She claimed that it would be better for her son to die if that was his fate. The pediatrician sought an order from the state court. The mother was present with her lawyer at the court. In deciding whether it may issue such an order in face of all relevant constitutional defenses by the mother, which of the following must the state court consider?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}