{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1db5dda4-f7a7-5c9e-85d2-ed25788706fe","task_key":"all--test--1db5dda4-f7a7-5c9e-85d2-ed25788706fe","task_revision_id":"1","upstream_id":"","short_description":"Apart from the fact that it is \"not a nice thing to do\", what is an important…","config":"all","split":"test","body":"{\"choices\":\"['It can damage the professional reputation of the researcher and their discipline.', 'It makes it more difficult to gain access to deviant or hidden populations.', 'It means that records of personal data about the participants cannot be made anonymous.', 'None of the above.']\",\"question\":\"Apart from the fact that it is \\\"not a nice thing to do\\\", what is an important ethical disadvantage of deceiving participants?\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}