{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ebe980bf-64f5-567c-bf86-9d9fc9292c53","task_key":"default--test--1627","task_revision_id":"1","upstream_id":"1627","short_description":"A company created a new brand of pies. However, a study by the federal Food and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"constitutional, because it is within the state's police power.\\\", 'unconstitutional, because the federal law takes precedence in matters of public safety.', 'constitutional, because the state has a compelling interest in protecting its citizens.', 'unconstitutional, because it infringes on the rights of the pie company.', 'constitutional, because Congress did not expressly preempt state legislation.', 'constitutional, because the state has the right to regulate all commerce within its borders.', 'constitutional, because Congress may not regulate an economic activity where both buyer and seller reside in the same state.', 'unconstitutional, because it affects interstate commerce.', 'unconstitutional, because the state cannot contradict federal regulations.']\",\"question\":\"A company created a new brand of pies. However, a study by the federal Food and Drug Administration revealed that the pies contain potentially harmful levels of nuts for some consumers with nut allergies. As a result, Congress enacted legislation prohibiting the shipment and sale of the pies across state lines. A state has a statute that regulates the shipment and sale of the pies within its territory. In light of the federal legislation prohibiting the shipment and sale of the pies across state lines, the state statute is probably\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}