{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4058cfe6-c9a8-5535-8613-04f04e35e4f1","task_key":"all--test--4058cfe6-c9a8-5535-8613-04f04e35e4f1","task_revision_id":"1","upstream_id":"","short_description":"A ski resort had a lift that carried customers from the bottom to the top of the…","config":"all","split":"test","body":"{\"choices\":\"['false imprisonment.', 'negligence.', 'assault.', 'no cause of action.']\",\"question\":\"A ski resort had a lift that carried customers from the bottom to the top of the ski slope. One afternoon, the lift suddenly broke down because of a power failure in the area. A customer was suspended in his lift chair, one-third of the way up the hill and 50 feet above the ground. The customer remained on the lift for five hours until power was restored. He was then returned uninjured to the bottom of the hill. In a suit against the ski resort, the customer is likely to have action for\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}