{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2061e2cb-92c8-5d63-8a9b-a1a526cb32cf","task_key":"all--test--2061e2cb-92c8-5d63-8a9b-a1a526cb32cf","task_revision_id":"1","upstream_id":"","short_description":"A husband visits an attorney seeking a divorce because of his wife's infidelity.…","config":"all","split":"test","body":"{\"choices\":\"['admissible, because her presence during the meeting destroyed the attorney-client privilege.', \\\"admissible, because the law clerk's presence during the meeting destroyed the attorney- client privilege.\\\", 'inadmissible, because the attorney-client privilege disqualifies the secretary from testifying to such confidential communications.', 'inadmissible, because it is hearsay not within any recognized exception.']\",\"question\":\"A husband visits an attorney seeking a divorce because of his wife's infidelity. At the client consultation, the attorney's secretary took notes. With the permission bf the husband, the attorney's law clerk also sat in on the consultation to see how the attorney conducted these meetings. Shortly after this initial consultation with the attorney, the secretary quit. During the divorce trial, the wife's attorney seeks to call the secretary to testify about what the husband told the attorney during the initial client consultation. Upon objection, the secretary's proposed testimony will most likely be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}