{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"865b3e6e-9f7d-5496-855c-de4e94a01daa","task_key":"all--test--865b3e6e-9f7d-5496-855c-de4e94a01daa","task_revision_id":"1","upstream_id":"","short_description":"A research report states that a special education program for children aged 3-5…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Children's cognitive abilities are constantly changing at the age of 3-5\\\", 'Establishing such education and training programs on a national basis requires a special public expenditure', 'Many parents mistakenly believe that early formal education will occupy the time that children would have been able to better explore the world independently', 'Investigators are unaware that they include a large group of children who have previously received another education']\",\"question\":\"A research report states that a special education program for children aged 3-5 under study increases their chances of success in future schooling. Therefore, implementing a similar education program for all children will improve them in the future opportunities for success in school education. Which of the following best illustrates the logical loopholes summarized above?\",\"subject\":\"miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}