{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8c2441be-ec86-552a-8966-802a7347a6c7","task_key":"all--test--8c2441be-ec86-552a-8966-802a7347a6c7","task_revision_id":"1","upstream_id":"","short_description":"A homeowner had a beautiful beach house overlooking the bay. Although the…","config":"all","split":"test","body":"{\"choices\":\"['voluntary manslaughter.', 'involuntary manslaughter.', 'murder.', 'assault with a deadly weapon.']\",\"question\":\"A homeowner had a beautiful beach house overlooking the bay. Although the homeowner and his family lived in the beach house during the summer months, the house was left unoccupied for the remainder of the year. In order to protect the beach house from vandalism while it was vacant, the homeowner installed an automatic spring-gun device. The spring-gun was connected to the front door and calculated to automatically fire at chest level when the door was opened. No warnings were placed on the premises. A local man, knowing that the homeowner's beach house was unoccupied, decided to burglarize the home one evening. When the local man forced open the front door and was about to enter the premises, the spring-gun automatically discharged, killing him instantly. If the homeowner is subsequently prosecuted and charged with the local man's death, the most serious crime for which the homeowner will be found guilty is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}