{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"063c7982-8209-55e8-b3fb-0fae7b638258","task_key":"all--test--063c7982-8209-55e8-b3fb-0fae7b638258","task_revision_id":"1","upstream_id":"","short_description":"At a metro station in the city, there is always significant construction going…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because she was fully informed and should have known that the elevator was unsafe based upon the signs.', 'Yes, because the signs make her contributorily negligent and a legal cause of her own injury.', 'No, because the sign did not explain the danger and only told her that it was not available, and she therefore did not know what she was encountering.', 'No, because the city has strict liability when everyone is depending on the subway.']\",\"question\":\"At a metro station in the city, there is always significant construction going on, including a rebuild of the elevator shaft between the platform and the ground level. There is a sign warning \\\"this elevator is not available at this time, please do not ride it.\\\" A passenger sees the signs, but tries to ride the elevator anyway, and is injured when she falls into the shaft. When the passenger sues the city for negligence, the city states that she assumed the risk. Is that a valid and accurate defense under the circumstances?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}