{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"76a07399-da10-56d5-acf3-61199e4f497f","task_key":"all--test--76a07399-da10-56d5-acf3-61199e4f497f","task_revision_id":"1","upstream_id":"","short_description":"Dr. Mansfield Mayhem receives a subpoena requiring him to testify about a…","config":"all","split":"test","body":"{\"choices\":\"['notify the attorney who issued the subpoena that the client has asserted the privilege and request that he be released from subpoena.', 'do nothing further until he receives an order from the court to appear at the trial.', 'notify the court that he will not be appearing at the trial because the client does not want him to do so.', 'advise the attorney who issued the subpoena that he cannot appear as requested because the client has not given him permission to do so.']\",\"question\":\"Dr. Mansfield Mayhem receives a subpoena requiring him to testify about a current therapy client at a trial. Dr. Mayhem contacts the client who states that she does not want him to release any confidential information to the court. Dr. Mayhem should:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}