{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ad230ebe-f0c8-529d-819e-b6ec113ad337","task_key":"all--test--ad230ebe-f0c8-529d-819e-b6ec113ad337","task_revision_id":"1","upstream_id":"","short_description":"Following the retirement of his business partner, Dr. Sheldon Smythe is the only…","config":"all","split":"test","body":"{\"choices\":\"['refuse to see the clients.', 'see these clients but use only interventions that he has experience using.', 'see the clients and obtain supervision or consultation by telephone.', 'agree to see these clients only if they are experiencing a crisis.']\",\"question\":\"Following the retirement of his business partner, Dr. Sheldon Smythe is the only psychologist in a small town, and the next closest mental health professional works in a town that is over 100 miles away. Several of Dr. Smythe’s new clients have problems for which he has limited training and experience. Dr. Smythe should:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}