{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"94f19d02-4188-563a-88d9-6cfdc90d5dc1","task_key":"all--test--94f19d02-4188-563a-88d9-6cfdc90d5dc1","task_revision_id":"1","upstream_id":"","short_description":"In which situation would the defendant most likely be guilty of murder?","config":"all","split":"test","body":"{\"choices\":\"['As a practical joke, the defendant throws a firecracker in a crowded movie theater. A victim, who has a weak heart, becomes frightened and thinks someone is firing a gun. The victim suffers a heart attack and dies.', 'The defendant hates her husband after learning that he is having an affair with her best friend. She also knows that her husband owes a man a large amount of money, which he refuses to pay. Before the husband is about to leave for work, the defendant looks out the window and sees the man hiding outside with a gun. The defendant decides not to warn the husband, who then walks outside and is shot and killed by the man.', \\\"The defendant and his friend attended a party together. At the party, the friend became extremely inebriated. The friend asked the defendant if she could borrow his car to drive to the liquor store to buy more booze for the party. Although the defendant was aware of the friend's intoxicated condition, he allowed her to drive his car to the liquor store. The friend drove through a red light and collided with the victim's car. The victim was killed in the accident.\\\", \\\"The defendant keeps a loaded gun in his house for protection. One day, the defendant leaves the gun on the kitchen table and leaves for work. That same day, a robber breaks into the defendant's home, sees the gun, and steals it. The robber then uses the gun to rob a convenience store. During the robbery, the robber shoots and kills the proprietor with the defendant's gun.\\\"]\",\"question\":\"In which situation would the defendant most likely be guilty of murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}