{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"010cbf4a-dbbc-5fe8-bb6c-5d049dabda34","task_key":"all--test--010cbf4a-dbbc-5fe8-bb6c-5d049dabda34","task_revision_id":"1","upstream_id":"","short_description":"A first-class passenger on an airplane flight from Atlanta to Chicago was seated…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because, as a first-class passenger, she was owed a special duty of care by the airline.', 'prevail, because the flight attendant should have been aware that her conduct caused an unjustifiable risk of harm.', 'not prevail, because the airline is not vicariously liable for the tortious conduct of its passengers.', 'not prevail, because the passenger assumed the risk by not moving to another seat away from the salesman.']\",\"question\":\"A first-class passenger on an airplane flight from Atlanta to Chicago was seated next to a middle-aged salesman who was returning home after attending a business convention in Atlanta. The passenger, who was extremely exhausted after a long and hectic day, tried to sleep during the flight but was constantly being annoyed by the salesman. Once the flight departed, the salesman started ordering drinks of Scotch and water from the flight attendant, and became inebriated. When the passenger refused to talk to him, the salesman became very abusive to the passenger. Although there were many empty seats available on the plane in the first-class section, the passenger decided to remain seated next to the salesman. Finally, after the flight attendant had served the saleman his tenth drink of Scotch, he became belligerent and punched the passenger in the mouth. The passenger's two front teeth were knocked loose and she suffered a cut lip. If the passenger asserts a claim against the airline based on negligence, she will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}