{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1b9688f-f109-59e0-94ce-a4d5a0c77fd0","task_key":"all--test--d1b9688f-f109-59e0-94ce-a4d5a0c77fd0","task_revision_id":"1","upstream_id":"","short_description":"A retired cattle rancher was flying his private plane to visit his daughter and…","config":"all","split":"test","body":"{\"choices\":\"['recover, because the property belonged to a private person.', 'recover, because the rancher had reasonable grounds to believe that the orange grove belonged to a private person.', 'not recover, because the rancher was privileged to enter the land of another under the circumstances.', 'not recover, because the rancher was carefully and skillfully operating his airplane when the engine went out.']\",\"question\":\"A retired cattle rancher was flying his private plane to visit his daughter and grandchildren. While carefully and skillfully operating his airplane, he suddenly realized that one of his engines was gone. He was then forced to make an emergency landing under the reasonable belief that it was necessary to do so for the protection of himself and his airplane. He landed in a farmer's orange grove and managed to steer the plane and land it between the trees in order to not cause damage to any of them. If the farmer brings suit against the rancher for trespass, the most likely result is that the farmer will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}