{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a0ebdb3a-cb9d-50be-ae3a-a2236df453b5","task_key":"all--test--a0ebdb3a-cb9d-50be-ae3a-a2236df453b5","task_revision_id":"1","upstream_id":"","short_description":"Dr. Carl Carlson is a licensed psychologist and professor in the psychology…","config":"all","split":"test","body":"{\"choices\":\"['clearly violating ethical guidelines.', 'acting ethically since he is not the one who initiated a potential relationship.', 'acting ethically as long as Marla is a student in a department other than the psychology department and will not be taking any classes in the psychology department.', 'acting ethically as long as he does not become sexually involved with Marla until she is no longer affiliated with the university.']\",\"question\":\"Dr. Carl Carlson is a licensed psychologist and professor in the psychology department at a large university. He is approached by a graduate student, Marla M., age 36, who asks him if he’d like to attend a baseball game on Friday night. Dr. Carlson is divorced and finds Marla to be a very attractive woman. In terms of ethical guidelines, if Dr. Carlson accepts Marla’s invitation, he will be:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}