{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"81bf3f68-75b4-5c32-9727-c423a7e9b05a","task_key":"all--test--81bf3f68-75b4-5c32-9727-c423a7e9b05a","task_revision_id":"1","upstream_id":"","short_description":"A pilot decided to fly his small airplane to another city for a weekend visit.…","config":"all","split":"test","body":"{\"choices\":\"['recover under strict liability, because the pilot engaged in an abnormally dangerous activity.', 'recover under negligence, because the pilot failed to exercise reasonable care in the operation of his airplane.', 'not recover, because the passenger did not pay to go on the flight.', 'not recover, because the passenger unreasonably exposed himself to a known danger.']\",\"question\":\"A pilot decided to fly his small airplane to another city for a weekend visit. He invited one of his best friends to join him as a passenger on the trip. Near the end of the flight, the pilot saw a cloud of fog ahead but decided that he could navigate through it. He got lost in the fog and crashed the plane into the side of a mountain. The pilot and his passenger were both killed instantly in the crash. If the passenger's family brings a wrongful death action against the pilot's estate, the most likely result is that the plaintiffs will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}