{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"65a280b9-ec4c-5a4d-8cdf-cebe69f3ad39","task_key":"default--test--1618","task_revision_id":"1","upstream_id":"1618","short_description":"A pharmaceutical company manufactured a new contraceptive sponge. The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['constitutional, because Congresshas the power to regulate interstatecommerce.', \\\"unconstitutional, because it limits the pharmaceutical company's right to free trade.\\\", 'unconstitutional, because it deprives the manufacturer a property right without just compensation.', \\\"constitutional, because it falls under Congress's power to regulate harmful substances.\\\", 'unconstitutional, because it infringes on the rights of those with preexisting health conditions.', 'constitutional, because Congress has the power to provide for the general welfare.', 'constitutional, because Congress has the power to enact laws in response to public health concerns.', 'constitutional, as Congress has the power to regulate pharmaceutical products.', 'unconstitutional, because it interferes with the right of privacy of contraceptive users.', 'unconstitutional, because it restricts the freedom of choice for contraceptive users.']\",\"question\":\"A pharmaceutical company manufactured a new contraceptive sponge. The pharmaceutical company initially intended for the contraceptive to be available for purchase without a doctor's prescription. However, a study by the Food and Drug Administration revealed that the contraceptive might prove harmful to some users with preexisting health problems. As a result, Congress enacted legislation prohibiting the shipment and sale of the contraceptive across state lines to those who do not have a doctor's prescription. This law is probably\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}