{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d286baed-0666-578d-a2fe-31ea3ef42d45","task_key":"default--test--1852","task_revision_id":"1","upstream_id":"1852","short_description":"A devastating earthquake struck a foreign country. The earthquake destroyed…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['valid, because the President can repurpose federal agencies for emergency relief efforts.', \\\"valid, because the president's executive power extends to directing federal agencies in times of crisis.\\\", 'invalid, because the executive order did not specify the scope of the relief operations.', 'invalid, because the executive order infringes upon the rights of federal employees.', 'invalid, because the director of the INS, not the President, has the power to issue directives to federal agencies under its supervision.', 'valid, because the President has the power to issue executive orders in times of emergency.', 'invalid, because the executive order is beyond the scope of presidential power absent congressional authorization.', 'valid, because employees of an executive agency can be required to perform such emergency functions.', 'invalid, because the INS does not have jurisdiction in foreign countries.', 'valid, because as commander-in-chief, the President has unfettered authority to require federal employees to participate in an international relief effort.']\",\"question\":\"A devastating earthquake struck a foreign country. The earthquake destroyed hundreds of homes and buildings. Thousands of residents were killed, injured, and left homeless by the earthquake. The leader of the foreign country sought the assistance of the United States in providing emergency aid. The President initially wanted to send troops from a west coast military base but learned that they were out on training maneuvers. As a result, the President issued an executive order directing members of a federal agency under the supervision of the Immigration and Naturalization Service, to travel to the foreign country to engage in emergency rescue operations. This federal agency's main function is to patrol the borders to prevent illegal aliens from entering the United States. A member of this federal agency refused to comply with the executive order and assist in the relief effort. He claimed that participating in such operations in a foreign country is beyond the scope of his job responsibilities. He brought suit in federal court challenging the validity of the President's executive order. Assuming the member of the federal agency has adequate standing, the court should rule the executive order \",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}