{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c7ef10cb-df93-5645-ab72-8e006a130b3a","task_key":"all--test--c7ef10cb-df93-5645-ab72-8e006a130b3a","task_revision_id":"1","upstream_id":"","short_description":"A husband and wife were unhappily married. The husband was an alcoholic who…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because she purchased the life insurance policy.', 'guilty, because she intended to kill her husband.', 'not guilty, because she did not perform a substantial step in carrying out the murder.', 'not guilty, because she effectively withdrew from the criminal endeavor.']\",\"question\":\"A husband and wife were unhappily married. The husband was an alcoholic who physically abused the wife when he became drunk. After the husband beat up the wife one night, she decided to kill him. The wife, who was experienced in firearms, planned to shoot her husband while he slept. Before carrying out her plan, the wife took out a $1,000,000 life insurance policy on her husband, naming herself as beneficiary. Shortly thereafter, the husband and wife celebrated their 10th wedding anniversary. They went out to dinner and had a truly enjoyable evening together. After a great night, the wife experienced a change of heart and decided she really loved her husband and didn't want to kill him after all. The wife confessed and told her husband about her plan, begging forgiveness. The husband was outraged. He proceeded to file for divorce and notified the police about his wife's plan. If the wife is charged with attempted murder, she should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}