{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2a597c2a-c057-510b-a649-301e1f651cc4","task_key":"all--test--2a597c2a-c057-510b-a649-301e1f651cc4","task_revision_id":"1","upstream_id":"","short_description":"A professional football player was seriously injured when the football helmet he…","config":"all","split":"test","body":"{\"choices\":\"['admissible as an admission.', \\\"admissible as circumstantial evidence that the player's injury was traceable to the defective design of the football helmet.\\\", 'inadmissible, because of the public policy behind taking safety precautions.', 'inadmissible, because the former employee is no longer authorized to speak on behalf of the manufacturer.']\",\"question\":\"A professional football player was seriously injured when the football helmet he was wearing shattered as he was being tackled during a game. The player subsequently asserted a claim against the manufacturer of the helmet to recover damages for his injury. At trial, the player's attorney calls a former employee of the manufacturer to testify that three weeks after the player suffered his injury, the manufacturer devised a new design for its football helmets. The former employee's testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}