{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4561ae16-7d47-5333-badc-e03f15cc52b6","task_key":"all--test--4561ae16-7d47-5333-badc-e03f15cc52b6","task_revision_id":"1","upstream_id":"","short_description":"A woman became pregnant at a time when there were escalating marital problems…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, the law did not intrude on the wife's right to privacy because the husband had an equal right to be protected.\\\", \\\"No, the law presented only a minimum intrusion and it would not be difficult for married women to obtain their husband's consent.\\\", 'Yes, it is unconstitutional because a woman has an unqualified right to an abortion on demand.', \\\"Yes, it is unconstitutional because it presents an undue burden on a woman's right to obtain an abortion.\\\"]\",\"question\":\"A woman became pregnant at a time when there were escalating marital problems with her husband. She didn't tell him that she was pregnant, and decided to get an abortion without telling him. Her state had a law requiring her to sign a statement verifying that she had told her husband of her intended abortion. She refused to sign and sued the state to challenge the constitutionality of the law. Is the law unconstitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}