{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7b9e1d89-e855-529d-abd3-10c2d87a70bf","task_key":"all--test--7b9e1d89-e855-529d-abd3-10c2d87a70bf","task_revision_id":"1","upstream_id":"","short_description":"One Friday afternoon, just before leaving on a weekend tip, a psychologist…","config":"all","split":"test","body":"{\"choices\":\"[\\\"unethical, because the client's needs were not addressed more directly\\\", 'unethical, because the psychologist should have delayed the trip and given the client an appointment for that afternoon', 'unethical, because the psychologist should have called a physician directly', 'ethical, because it was not possible to determine whether or not the client was being manipulative']\",\"question\":\"One Friday afternoon, just before leaving on a weekend tip, a psychologist receives a call from a therapy client who claims to have taken an overdose of pills in a suicide attempt. The psychologist: tells the client to call a physician immediately and to come in at 9:00 A.M. on Monday for a session. The psychologist makes no other attempt to intervene, and the client dies later that evening without making any other calls for assistance. In this situation, the psychologist's behavior was clearly\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}