{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e7f7d260-d103-56f6-8905-66e899e298e9","task_key":"all--test--e7f7d260-d103-56f6-8905-66e899e298e9","task_revision_id":"1","upstream_id":"","short_description":"The prime minister of a foreign country was assassinated by a group of right…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the President, as commander- in-chief of the armed forces, has the power to authorize such military expeditions.', 'Yes, because the President has inherent power to regulate foreign affairs as long as there is no formal declaration of war.', \\\"No, because the President cannot usurp Congress's spending power by taking action that supersedes a congressional appropriations bill.\\\", 'No, because the executive order was neither authorized by federal statute nor ratified by the Senate.']\",\"question\":\"The prime minister of a foreign country was assassinated by a group of right wing extremists. The prime minister's death triggered a civil war between rival factions within the country. To prevent U. S. involvement in this country's political crisis, Congress passed an appropriations bill prohibiting any funding for U. S. military operations in this foreign country. The appropriations bill was passed over a presidential veto. Thereafter, the President issued an executive order directing U. S. Navy and Army troops to the foreign country to restore order. Is this executive order constitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}