{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"85946efd-228c-5a51-94dc-993a14f9b046","task_key":"all--test--85946efd-228c-5a51-94dc-993a14f9b046","task_revision_id":"1","upstream_id":"","short_description":"A homeowner awoke one night and observed a thief stealing chickens from his…","config":"all","split":"test","body":"{\"choices\":\"['Yes, he was authorized to use deadly force where he reasonably believed that the thief had a gun and would harm him.', 'No, because it is not reasonable to use deadly force to prevent the theft of the chickens.', 'No, because a homeowner can never legally use deadly force to stop a theft.', 'Yes, because a dwelling owner can always use deadly force to protect his property and person from an illegal trespasser or thief.']\",\"question\":\"A homeowner awoke one night and observed a thief stealing chickens from his backyard chicken coop. He thought he saw a gun but could not tell for sure. While the thief was exiting with a sack of chickens, the homeowner got his hunting rifle and shot him dead. The authorities charged the homeowner with homicide. At the trial, he defended on the basis that he was in fear for his safety, and was legally authorized to use deadly force to defend his dwelling and person from the illegal threat. Is the defense to homicide valid under these facts?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}