{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3ca36ab-8841-5165-a567-406a9991e34f","task_key":"all--test--e3ca36ab-8841-5165-a567-406a9991e34f","task_revision_id":"1","upstream_id":"","short_description":"An owner ran a sporting goods store that specialized in hunting and camping…","config":"all","split":"test","body":"{\"choices\":\"['recover, because the friend was negligent.', 'recover, because he feared the friend would hit him.', 'not recover, because he suffered no physical harm.', 'not recover, because the friend was only joking.']\",\"question\":\"An owner ran a sporting goods store that specialized in hunting and camping supplies. His friend had often cautioned him that he should not leave so much of his merchandise out in the open. One day, as the friend entered the store to pay a visit, he decided he'd put his unheeded warnings into action by playing a joke on the owner. The friend took a hatchet that was lying on a display case and swung it over his head as if he were going to strike the owner, who was standing at the register with his back turned. The friend then said, \\\"All right buddy, one false move and it's over. \\\" The owner was very frightened at almost being hit with the hatchet but suffered no actual physical or other harm. If the owner asserts a claim against the friend, the most likely result is that he will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}