{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e5956852-d447-5c06-a903-c69f7e1b4d17","task_key":"all--test--e5956852-d447-5c06-a903-c69f7e1b4d17","task_revision_id":"1","upstream_id":"","short_description":"A security guard had a license to carry a concealed handgun. One day, he was…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the security guard knew the barber was present.', 'prevail, because the security guard assaulted the friend.', \\\"not prevail, because the barber was not a member of the friend's immediate family.\\\", 'not prevail, because the resulting harm was unforeseeable.']\",\"question\":\"A security guard had a license to carry a concealed handgun. One day, he was walking past a barber shop when he saw his friend inside getting a haircut. The security guard knew that the barber, an elderly man, had a history of significant heart problems. The security guard decided to play a joke on his friend. The security guard took out his gun, burst through the door of the barber shop, pointed the gun at the friend, and shouted, \\\"Don't move or I'll shoot!\\\" The barber, thinking that an armed intruder was about to shoot the friend, suffered a heart attack and collapsed. If the barber brings suit against the security guard for infliction of emotional distress and seeks recovery for damages arising from the heart attack, he should\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}