{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"652e5086-4417-57db-8744-9c8f64fdc020","task_key":"all--test--652e5086-4417-57db-8744-9c8f64fdc020","task_revision_id":"1","upstream_id":"","short_description":"Believing she was pregnant, a woman went to see a doctor, a licensed…","config":"all","split":"test","body":"{\"choices\":\"['murder and larceny by trick.', 'manslaughter and false pretenses.', 'battery and larceny by trick.', 'murder, battery, and false pretenses.']\",\"question\":\"Believing she was pregnant, a woman went to see a doctor, a licensed gynecologist, because she wanted an abortion. The doctor examined her and determined that she was not pregnant, but decided that he would tell her she was, in order to earn his $500 fee. After receiving the $500, the doctor proceeded to have the woman admitted to a hospital by falsely informing the hospital authorities that she had a benign tumor on her uterus, which he was going to remove. He performed all the surgical procedures appropriate for an abortion under adequate hygienic conditions, but the woman began hemorrhaging and died. The doctor should be found guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}