{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11f0a468-e6df-5acc-b97e-91c7f2bf182d","task_key":"all--test--11f0a468-e6df-5acc-b97e-91c7f2bf182d","task_revision_id":"1","upstream_id":"","short_description":"A repairman repaired damaged electrical power lines and replaced old, worn-out…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because he was attempting to rescue the repairman.', 'prevail, because a public utility company is strictly liable in tort.', 'not prevail, because he was negligent in climbing the ladder.', 'not prevail, because the repairman was not negligent in not affixing the ladder properly.']\",\"question\":\"A repairman repaired damaged electrical power lines and replaced old, worn-out equipment whenever necessary for the local municipal electrical company. After a violent tornado had ripped through the city, the electrical company was busily trying to restore electrical power to its customers. The morning after the storm, the repairman was perched on a ladder trying to repair a high-voltage power line. As he was removing the cover of the transformer box, the ladder suddenly slipped on the wet ground, even though it had been properly fixed into position. The repairman struggled to maintain his balance and, luckily, the ladder came to rest against the transformer box. A pedestrian, who was walking on the street below, saw the repairman's predicament and began to climb the ladder to aid him. As the pedestrian was ascending the ladder, his foot slipped, and he fell to the ground. His fall caused the ladder to jar loose the transformer box, which in turn sent the repairman falling to the pavement. The repairman and the pedestrian both suffered serious injuries. If the pedestrian asserts a claim against the electrical company for his injuries, he will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}