{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"61335cc3-bf3d-5f53-85d8-91ddc3740abf","task_key":"all--test--61335cc3-bf3d-5f53-85d8-91ddc3740abf","task_revision_id":"1","upstream_id":"","short_description":"A common heart defect in humans is a ventricular septal defect, in which there…","config":"all","split":"test","body":"{\"choices\":\"['The percentage of oxygen in the left atrium would be lower.', 'The percentage of CO2 in the right ventricle would be higher.', 'The percentage of oxygen in the right atrium would be lower.', 'The percentage of oxygen in the right ventricle would be higher.']\",\"question\":\"A common heart defect in humans is a ventricular septal defect, in which there is a hole in the septum between the right and left ventricles. If a patient were to have this defect, which of the following statements is correct?\",\"subject\":\"college_medicine\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}