{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e1957105-e646-5961-8a9a-dc8ac7190241","task_key":"default--test--1742","task_revision_id":"1","upstream_id":"1742","short_description":"A man was arrested and convicted of publishing paid newspaper advertisements…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, the statute violates the First Amendment right to publish protected commercial speech.', 'Yes, the right to an abortion does not include the right to publicize where to get one.', 'Yes, the statute properly restricts speech that promotes unlawful activity.', 'No, the statute violates the Fourth Amendment protection against unreasonable searches and seizures.', 'No, the statute violates the Fifth Amendment right to due process.', 'No, the statute violates the sixth amendment right to privacy.', 'No, the statute violates the Eighth Amendment prohibition of cruel and unusual punishment.', 'Yes, the right to an abortion does not include the right to advertise one.', 'Yes, the statute does not violate any constitutional rights.', 'Yes, there is no right to publish or disseminate speech that is commercial in nature.']\",\"question\":\"A man was arrested and convicted of publishing paid newspaper advertisements giving information on how to get an abortion in another state and how to find a referral service. A criminal statute made it a crime to disseminate such information. Will the courts uphold the conviction?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}