{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"898f8e85-f0c8-5563-94d6-d02d971452a9","task_key":"all--test--898f8e85-f0c8-5563-94d6-d02d971452a9","task_revision_id":"1","upstream_id":"","short_description":"A pilot was flying his small plane when he experienced engine trouble and was…","config":"all","split":"test","body":"{\"choices\":\"['No, because the homeowner had provided adequate warning.', 'No, because the pilot was a trespasser.', 'Yes, because the pilot can invoke the privilege of necessity.', 'Yes, because the pilot could not reasonably have been expected to see the warning signs posted by the homeowner.']\",\"question\":\"A pilot was flying his small plane when he experienced engine trouble and was forced to make an emergency landing. He landed the plane safely in a large yard behind a home located in a relatively remote area. Unfortunately, when he disembarked from the plane, he was attacked and injured by two large dogs kept by the homeowner to discourage trespassers. The homeowner, who had seen the plane land, had ordered the dogs to attack. Several months earlier, the homeowner had posted large signs around the perimeter of the yard warning of the dogs. Does the pilot have a viable claim against the homeowner for battery?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}