{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cfb75685-8c27-583b-8c53-91ecf00f4789","task_key":"all--test--cfb75685-8c27-583b-8c53-91ecf00f4789","task_revision_id":"1","upstream_id":"","short_description":"Dr. Ryan is a psychotherapist in a small town. She receives a phone call from a…","config":"all","split":"test","body":"{\"choices\":\"['see the man only if she determines that their past relationship will not interfere with her objectivity.', 'see the man only if their relationship ended more than two years ago and she determines that it will not interfere with her objectivity.', 'see the man but discuss the potential for conflicts with him before beginning treatment.', 'refer the man to a colleague.']\",\"question\":\"Dr. Ryan is a psychotherapist in a small town. She receives a phone call from a man she was sexually involved with several years ago. The relationship lasted only three months and ended amicably. The man is now married and is having problems unrelated to their past relationship. He has called to see if he can begin seeing Dr. Ryan in therapy. Dr. Ryan should:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}