{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1f0a32e8-2ed9-53d3-8f60-b1ac25997d96","task_key":"all--test--1f0a32e8-2ed9-53d3-8f60-b1ac25997d96","task_revision_id":"1","upstream_id":"","short_description":"A client whom Dr. Oliver Olivetti has been seeing for several months has…","config":"all","split":"test","body":"{\"choices\":\"['ethically since he is taking the client’s welfare into consideration.', 'ethically only if not to do so would require the client to quit therapy.', 'ethically only if the insurance forms do not require him to sign a statement saying that treatment began after the effective date of the insurance.', 'unethically.']\",\"question\":\"A client whom Dr. Oliver Olivetti has been seeing for several months has recently changed jobs, and the client has learned that her new insurance plan does not begin covering pre-existing conditions for 12 months. The client asks Dr. Olivetti if he could write his bills so that it appears that she has just started therapy. If Dr. Olivetti complies with the client's request, he is acting:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}