{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"55a12294-173c-5927-a738-5388fe8e6233","task_key":"all--test--55a12294-173c-5927-a738-5388fe8e6233","task_revision_id":"1","upstream_id":"","short_description":"Historically, what has been the general pattern of defense spending in the…","config":"all","split":"test","body":"{\"choices\":\"['Steady increases over time.', 'Increased spending during wars, with demobilization bringing spending down but not back to prewar levels', 'Unpredictable fluctuations.', 'Increased spending during wars, with further increases after the war to prepare for future engagements.']\",\"question\":\"Historically, what has been the general pattern of defense spending in the United States?\",\"subject\":\"us_foreign_policy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}