{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11a6c8f8-b6f6-59d1-b10d-a598278a22cb","task_key":"default--test--5199","task_revision_id":"1","upstream_id":"5199","short_description":"Historically, what has been the general pattern of defense spending in the…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"[\\\"Defense spending is directly proportional to the country's GDP.\\\", 'Steady increases over time.', 'Increased spending during wars, with demobilization bringing spending down but not back to prewar levels', 'Decreased spending during wars, with increased spending during peacetime.', 'Massive increase in defense spending after World War II, with steady decreases since then.', \\\"Minimal spending on defense due to the country's neutral stance.\\\", 'Increased spending during wars, with further increases after the war to prepare for future engagements.', 'Steady decreases over time.', 'Unpredictable fluctuations.', 'Increased spending during wars, with spending returning to prewar levels after demobilization.']\",\"question\":\"Historically, what has been the general pattern of defense spending in the United States?\",\"src\":\"ori_mmlu-us_foreign_policy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}