{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"55393c34-9e1b-5a84-859c-6774422d66d7","task_key":"all--test--55393c34-9e1b-5a84-859c-6774422d66d7","task_revision_id":"1","upstream_id":"","short_description":"A breeder and owner of vicious guard dogs trained his dogs to attack strangers…","config":"all","split":"test","body":"{\"choices\":\"['murder.', 'involuntary manslaughter.', 'voluntary manslaughter.', 'reckless endangerment.']\",\"question\":\"A breeder and owner of vicious guard dogs trained his dogs to attack strangers at night. He often sold and leased his guard dogs to various business and factory owners who used the guard dogs to frighten away intruders from entering their premises at night. One evening, the breeder was in the back yard of his home training three of his guard dogs. The back yard was enclosed with a chain link fence and a latched gate that prevented the dogs from running out. After the training session, the breeder opened the gate and permitted the dogs to run loose in his front yard. Minutes later, a man was walking along the sidewalk in front of the breeder's house when he was attacked by one of the dogs. The man suffered severe injuries and died as a result of the attack. The breeder should be found guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}