{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4ce46f87-e1c5-5c57-bfab-e28e887b5533","task_key":"default--test--1089","task_revision_id":"1","upstream_id":"1089","short_description":"A company created a drug to lower high blood pressure. The President issues an…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['invalid, because it is an infringement on the powers of the Food and Drug Administration.', 'valid, because the President can make decisions based on potential risks to public health.', 'valid, because the President has the authority to ensure that laws are faithfully executed.', 'invalid, because the President does not have the power to regulate interstate commerce.', 'invalid, because the President cannot base decisions on unproven scientific evidence.', 'valid, because the President has the authority to regulate potentially harmful substances.', 'valid, because the President has the authority to protect public health.', 'valid, because the President has the authority to impose economic regulations unless overruled by Congress.', 'invalid, because it infringes on the rights of the pharmaceutical company.']\",\"question\":\"A company created a drug to lower high blood pressure. The President issues an executive order prohibiting the shipment and sale of the drug within the United States. Under this executive order, federal agents are empowered to prosecute all interstate shippers and sellers of the drug. The President claims that he is acting upon conclusive evidence that the drug causes cervical cancer in laboratory monkeys. This executive order is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}