{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07b46ef2-106f-5dd9-aa98-094ea2febb6c","task_key":"all--test--07b46ef2-106f-5dd9-aa98-094ea2febb6c","task_revision_id":"1","upstream_id":"","short_description":"A woman was driving home after attending a concert. While talking on her…","config":"all","split":"test","body":"{\"choices\":\"['granted, because the accident would not have occurred but for the emergency.', 'granted, because the police officer assumed the risk of injury by answering the emergency call.', \\\"denied, because the police officer's injury was not related to any special danger of his job.\\\", \\\"denied, because the firefighter's rule does not apply to police officers.\\\"]\",\"question\":\"A woman was driving home after attending a concert. While talking on her cellphone, she momentarily took her eyes off the road. The woman then drove through a red light and struck a police car driven by a police officer, injuring him. At the time of the accident, the police officer was returning from an emergency dispatch. The police officer sued the woman to recover damages suffered in the auto accident. At trial, the police officer presented evidence that he was returning from an emergency scene when the woman drove through a red light and struck his patrol car. At the conclusion of the plaintiff's presentation of evidence, the woman moved for a summary judgment claiming the \\\"firefighter's rule\\\" barred recovery. The motion for summary judgment should be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}