{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"714aacc1-2fd5-5369-9e88-a38a031b89bb","task_key":"all--test--714aacc1-2fd5-5369-9e88-a38a031b89bb","task_revision_id":"1","upstream_id":"","short_description":"A psychologist who shares an office with another psychologist reruns around…","config":"all","split":"test","body":"{\"choices\":\"['speak with the colleague privately to resolve this incident informally', 'report the colleague to the appropriate local, state/provincial, or national ethics committee', 'say nothing further regarding this incident unless there is 2 complaint from the client', \\\"contact the colleague's client to offer assistance and/or professional advice\\\"]\",\"question\":\"A psychologist who shares an office with another psychologist reruns around midnight to retrieve some papers and unexpectedly comes upon the colleague engaging in sexual relations with one of that colleague's clients. According to ethics codes published by the American and Canadian Psychological Associations, the first psychologist should\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}