{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b2c98867-989a-5185-a267-7c69fc56b649","task_key":"all--test--b2c98867-989a-5185-a267-7c69fc56b649","task_revision_id":"1","upstream_id":"","short_description":"To save the expense of purchasing a new test form, a chief school administrator…","config":"all","split":"test","body":"{\"choices\":\"['administer the outdated form but use new norms', 'ask that at least a few copies of the new form be administered for comparison', 'insist on administering only the most recent form of the test', 'recommend that the assessment be canceled']\",\"question\":\"To save the expense of purchasing a new test form, a chief school administrator asks a certified school psychologist to administer a six-year-old form of an achievement test that agreed with the curriculum of the school district at that time. Considering the situation and the Principles for Professional Ethics of the National Association of School Psychologists' (NASP) the school psychologist should\",\"subject\":\"high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}