{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d447a102-a7b8-5603-8644-bf018412bbf9","task_key":"all--test--d447a102-a7b8-5603-8644-bf018412bbf9","task_revision_id":"1","upstream_id":"","short_description":"In which of the following situations would the defendant not be guilty of…","config":"all","split":"test","body":"{\"choices\":\"['A defendant came into a bar looking for a fight. He walked up to a victim, tapped him on the shoulder and said, \\\"You bother me. Get out of here. \\\" The victim ignored him, and the defendant proceeded to punch the victim in the face and stab him in the arm with a knife. The victim thereupon took out a knife that had been concealed in his pocket and stabbed the defendant in the right arm. The defendant, fearful that the victim would stab him in the heart, took out a gun and shot the victim to death.', \\\"A defendant was home in bed with a fever one night, and all the lights in his house were off. A victim, who was scouting the neighborhood that night for a house to burglarize, broke into the defendant's house through the basement window and went upstairs to the bedrooms to look for jewelry. The defendant, who was not aware that someone else was in the house, was startled when he saw the victim walk past his room toward the stairs leading to the outside doorway. The defendant pulled out a pistol from under his pillow when he saw the victim and shot him to death.\\\", 'A defendant started a joke about the victim\\\\'s brother. When word got to the victim about the defendant\\\\'s joke, the victim became incensed. He rushed to the defendant\\\\'s home, broke open the door and found the defendant preparing dinner in the kitchen. He immediately said, \\\"I\\\\'m going to kill you. \\\" The defendant knew that the victim had been convicted of attempted murder several years ago, and he cringed when the victim took out a gun and pointed it at him. The defendant could have easily darted for the open front door and evaded the victim but, instead, he suddenly pulled a knife from the kitchen wall, lunged at the victim, and stabbed him to death. Unknown to the defendant, the victim\\\\'s gun was not loaded.', 'A defendant was a reporter for a newspaper and was sent on assignment to another state to cover the story of a mining disaster. He was sitting in his hotel room one evening, trying to get a message to his editor, when a victim knocked at the door. The defendant opened the door, and the victim announced a robbery. The defendant took a lamp from the night table and threw it at the victim. The victim was momentarily stunned, and the defendant then pulled a gun out of his own suitcase and shot the victim to death.']\",\"question\":\"In which of the following situations would the defendant not be guilty of homicide?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}