{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"88f69379-f2f4-5835-8b1b-05e7f493b5f1","task_key":"all--test--88f69379-f2f4-5835-8b1b-05e7f493b5f1","task_revision_id":"1","upstream_id":"","short_description":"A neighborhood homeowners' association hired a security company to provide an…","config":"all","split":"test","body":"{\"choices\":\"[\\\"prevail, because the guard used unreasonable force to protect the homeowner's property.\\\", 'prevail, because the guard did not give a warning that he had a gun and was about to shoot.', \\\"not prevail, because the young man was trespassing on the homeowner's property.\\\", 'not prevail, because the young man was engaged in theft when he was shot.']\",\"question\":\"A neighborhood homeowners' association hired a security company to provide an armed guard to patrol the neighborhood. One evening, the guard saw a young man stealing a rake that a homeowner in the neighborhood had left outside in his yard. The guard ran toward the young man. Seeing the guard, the young man dropped the rake and began to flee. Believing that the young man was about to escape, the guard pulled out his gun and fired a shot at the young man. The bullet struck the young man in the arm. If the young man asserts a claim against the security company for damages for his injuries, the young man will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}