{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aec74c51-da5f-59a2-af72-7aa2f0d83a61","task_key":"default--test--1117","task_revision_id":"1","upstream_id":"1117","short_description":"Which of the following executive orders will most likely be found…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The President issued an executive order instructing all executive branch employees to refrain from using social media during working hours.', 'The President issued an executive order banning all forms of meat consumption within the premises of the executive building, citing health reasons and environmental concerns.', 'The President issued an executive order abolishing a federal agency that was established by an Act of Congress, citing redundancy and inefficiency.', \\\"In 1887, Congress passed a law establishing a federal Commission of Birdwatchers. The Commission, which is still in effect, consists of seven members who are appointed by the President. The Commission's main function is to go on periodic retreats to photograph and study North American birds and their migratory habits. Believing that the Commission is archaic, the President decides that any future funding will simply be a waste of money. He thus executes an executive decree abrogating the Commission of Birdwatchers.\\\", 'Without seeking the advice and consent of the Senate, the President issued an executive order to establish a new federal agency to oversee the operations of all other federal agencies.', 'The President issued an executive order requiring all executive branch employees to use exclusively one brand of ballpoint pens and pencils as their writing utensils. According to a study, the federal government could save in excess of $250,000 a year in office supply costs if all executive agencies were to use standardized pens and pencils.', 'The President issued an executive order declaring a state of emergency in a city dealing with civil unrest, and authorizes the use of military force without the approval of Congress.', 'The President issued an executive order mandating all executive branch employees to take a mandatory vacation for a period of one month.', \\\"The President issued an executive order requiring all executive branch employees to wear only white shirts and blouses during regular working hours. According to the President's directive, executive employees are prohibited from wearing colored (e. g. , blue or yellow) or striped shirts and blouses while on duty. The President issued the order in an effort to establish a uniform dress code for all executive employees.\\\", 'A devastating hurricane damages an island which is part of a foreign country. The storm destroys many homes, resulting in death and injury to thousands. In response to a request from the foreign government for emergency aid, the President, without seeking the advice and consent of the Senate, issues an executive decree authorizing U. S. Army troops to the island to provide medical and humanitarian assistance.']\",\"question\":\"Which of the following executive orders will most likely be found unconstitutional?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}