{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"533b82cf-5e7d-5949-b2ce-29ec177be2db","task_key":"all--test--533b82cf-5e7d-5949-b2ce-29ec177be2db","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"[\\\"President Thomas Jefferson's veto of the Alien and Sedition Acts.\\\", \\\"President James Monroe's veto of an act for the preservation and repair of the Cumberland Road.\\\", \\\"President Andrew Jackson's veto of the bill rechartering the Second Bank of the United States.\\\", \\\"President Andrew Johnson's veto of one of the Reconstruction Acts of 1867.\\\"]\",\"question\":\"This question refers to the following information.\\n\\\"The power . . . given to the commanding officer over all the people of each district is that of an absolute monarch. His mere will is to take the place of all law. . . . It reduces the whole population of the ten states—all persons, of every color, sex, and condition, and every stranger within their limits—to the most abject and degrading slavery.\\\"\\nThe excerpt from the presidential veto message above is from\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}