{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d6bc1f7d-3a8d-5864-9ecd-ca590d3a332f","task_key":"default--test--913","task_revision_id":"1","upstream_id":"913","short_description":"Because of a sudden and unanticipated severe shortage of heating fuel, the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the order relates to the management of the executive branch and is not prohibited by any statute.', 'Yes, because the President has the authority to modify operational days in case of emergencies.', 'No, because the President cannot change the operating days without a formal amendment to the appropriations statute.', 'Yes, because the appropriations statute does not specifically state the number of operational days.', \\\"No, because when they passed the statute appropriating monies for the operation of executive agencies, members of Congress assumed that those agencies' offices would be open five days per week.\\\", \\\"No, because the President's order infringes on the rights of federal employees.\\\", \\\"No, because the heads of the various executive agencies have final responsibility for the operation of those agencies' offices.\\\", 'No, because a sudden and severe shortage of heating fuel does not constitute an emergency.', 'Yes, because the Constitution vests the President with plenary authority to direct the administration of all federal agencies in any manner the President deems expedient.', \\\"Yes, because the President's order still allows for exceptions in emergency circumstances.\\\"]\",\"question\":\"Because of a sudden and unanticipated severe shortage of heating fuel, the President has ordered all offices of federal executive agencies to be open only four days per week. The President's order allows an exception to the extent that emergency circumstances require different hours of operation (as in the case of federal hospitals). When Congress enacted the appropriations statute for operating all federal executive agencies, its members assumed that the offices of those agencies would be open five days per week, but Congress did not include such a requirement in its appropriations statute or in any other statute. Is the President's order constitutional?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}