{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"78055552-9b4b-53eb-9bb8-b8295a15a6b2","task_key":"all--test--78055552-9b4b-53eb-9bb8-b8295a15a6b2","task_revision_id":"1","upstream_id":"","short_description":"A student participant in a research study involving matched pairs decides to…","config":"all","split":"test","body":"{\"choices\":\"['warn the student that withdrawal from the study will affect his course grade.', 'remind the student that he signed an informed consent and cannot now request that his data be withdrawn.', 'tell the student that his name will be removed from all data and include his data in the study.', 'allow the student to withdraw from the study.']\",\"question\":\"A student participant in a research study involving matched pairs decides to withdraw from the study. The chief investigator (who is a licensed psychologist) stresses to the student the importance of the student’s data to the study, but he insists that his data be withdrawn. The investigator should:\",\"subject\":\"professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}