{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"50a784f5-922d-5299-9f4c-1aee05122106","task_key":"all--test--50a784f5-922d-5299-9f4c-1aee05122106","task_revision_id":"1","upstream_id":"","short_description":"A man was on trial for murder. Following a recess, the man was in the hallway…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, because the man's statement was a confidential communication and protected under the attorney-client privilege.\\\", 'No, because a judge may not testify in a matter over which he or she is presiding.', 'Yes, because a judge has a duty to disclose incriminating evidence to the prosecution.', \\\"Yes, because the man's statement was not a confidential communication.\\\"]\",\"question\":\"A man was on trial for murder. Following a recess, the man was in the hallway outside the courtroom speaking with his attorney. During their conversation, the man said, \\\"So what if I killed him, big deal. \\\" The man's statement was overheard by the presiding judge as he was walking down the corridor to the rest room. The judge then informed the prosecuting attorney of the man's comment. After trial is reconvened, the prosecutor calls the judge as a witness to testify to what he heard the defendant tell his attorney. Upon objection by defendant's attorney, may the judge be called as a witness?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}