{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"161aba19-73c2-57ca-af37-f1d6f3fd7a45","task_key":"all--test--161aba19-73c2-57ca-af37-f1d6f3fd7a45","task_revision_id":"1","upstream_id":"","short_description":"A mail carrier was on a homeowner's premises delivering the mail when he was…","config":"all","split":"test","body":"{\"choices\":\"['No, he is not liable because he used due care and was not negligent under the circumstances.', \\\"No, he is not liable because the man was on someone else's property at the time of the incident.\\\", 'Yes, he is liable because a dog owner is always liable for any injuries caused to a third person.', \\\"Yes, he is liable because an owner who knows of the dog's dangerous propensities is strictly liable for ensuing damages.\\\"]\",\"question\":\"A mail carrier was on a homeowner's premises delivering the mail when he was bitten by a large dog owned by the next door neighbor. Although the owner used due care in keeping the animal in a securely fenced area, he knew of one prior incident where the animal got loose and bit someone. Is the dog's owner liable for the injuries to the mailman?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}