{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b50b65d3-3105-5f2f-bb98-5fa88e34beeb","task_key":"all--test--b50b65d3-3105-5f2f-bb98-5fa88e34beeb","task_revision_id":"1","upstream_id":"","short_description":"A state has the following homicide statute in effect:\"Whoever, purposely and…","config":"all","split":"test","body":"{\"choices\":\"['admissible.', 'admissible, but the jury must be cautioned that it does not pertain to manslaughter.', 'admissible, but the jury must be cautioned that it does not pertain to felony murder or seconddegree murder.', 'not admissible.']\",\"question\":\"A state has the following homicide statute in effect:\\\"Whoever, purposely and with premeditated malice, or in the perpetration of, or attempt to perpetrate a rape, arson, robbery, burglary, or any other felony dangerous to human life, kills any human being, is guilty of murder in the first degree, and, on conviction, shall suffer death or be imprisoned in the state prison for life; Whoever maliciously but without premeditation kills any human being, is guilty of murder in the second degree, and, on conviction, shall be imprisoned in the state prison for life; Whoever unlawfully kills any human being without malice, express or implied, either voluntarily upon a sudden heat or involuntarily, but in the commission of some unlawful act, is guilty of manslaughter, and on conviction, shall be imprisoned in the state prison not less than two years nor more than 20 years. \\\"A man is having an affair with a married woman. The husband finds out about his wife's relationship with the man. The next day, the husband sees the man walking down the street. Enraged by the man's relationship with his wife, the husband pulls out a gun and shoots the man, killing him. The husband is subsequently arrested and prosecuted under the homicide statute for killing the man. At trial, the husband's attorney attempts to introduce evidence that at the time the husband shot the victim, he was intoxicated. Upon objection by the prosecuting attorney, the evidence is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}