{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ec52e652-086d-5147-9983-983b6f021a6c","task_key":"all--test--ec52e652-086d-5147-9983-983b6f021a6c","task_revision_id":"1","upstream_id":"","short_description":"Late one afternoon, a woman was hitchhiking when she was picked up by a man.…","config":"all","split":"test","body":"{\"choices\":\"['admissible as an admission.', 'admissible as a statement against interest.', \\\"inadmissible, because it violates the woman's privilege against self-incrimination.\\\", 'inadmissible, because it violates the attorney-client privilege for confidentialcommunications.']\",\"question\":\"Late one afternoon, a woman was hitchhiking when she was picked up by a man. Shortly thereafter, the man stopped and parked his car in a roadside rest area. They were smoking marijuana when another car skidded and crashed into the man's car. The collision damaged the man's car and inflicted personal injuries upon him. The woman was likewise injured. In fact, the impact produced a state of unconsciousness in her that lasted several minutes. The woman sued the other driver seeking to recover for his alleged negligence. At trial, the woman testified to her injuries and to the other driver's negligence. In defense, the other driver called an attorney who lived next door to the woman. The attorney proposed to testify that after the accident, the woman consulted him about her claim and asked the attorney in confidence how she could falsely testify that she wasn't smoking marijuana at the time of the accident. Upon objection by the woman's attorney, the attorney's testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}