{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa05b391-dc9c-5e81-aa58-f6748b144cc6","task_key":"all--test--aa05b391-dc9c-5e81-aa58-f6748b144cc6","task_revision_id":"1","upstream_id":"","short_description":"A man was arrested under a criminal statute that prohibited public intoxication.…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because it is unconstitutional to punish someone for the status of being an alcoholic or a drug addict.', 'Yes, because when a person acts under a compulsion that is part of a medical disease, this negates criminal intent.', 'No, because the man was sane and knew the difference between right and wrong when he was sober.', 'No, because he was convicted not for being a chronic alcoholic but for the actions of being in public while drunk.']\",\"question\":\"A man was arrested under a criminal statute that prohibited public intoxication. A magistrate fined him $100 and sentenced him to a weekend in jail. On appeal, he had a de novo trial before a county trial court. He presented medical testimony diagnosing the disease of alcoholism, which was marked by a nearly uncontrollable urge to drink. The defense claimed that a person could not be convicted for suffering from a disease process. The state simply argued that the man had no defense to what he did. The U.S. Supreme Court accepted the man's appeal after his conviction. Will the Court likely reverse the conviction?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}