{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f357cd4a-8d95-5484-b8cf-35eba5c21e9c","task_key":"default--test--1358","task_revision_id":"1","upstream_id":"1358","short_description":"A state law prohibits any barbershop licensed by the state from displaying…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because it leaves political candidates free to communicate their campaign messages to voters by other means.', 'Yes, because it prevents potential bias towards a candidate in a professional establishment.', 'Yes, because the prohibition is limited to the display of campaign material and does not restrict other forms of political expression.', 'No, because it imposes a restriction on the content or subject matter of speech in the absence of any evidence that such a restriction is necessary to serve a compelling state interest.', 'Yes, because the law only applies to current candidates and not former or future ones.', 'Yes, because the operation of a licensed barbershop is a privilege and therefore is subject to any reasonable restriction imposed by the state.']\",\"question\":\"A state law prohibits any barbershop licensed by the state from displaying posters in support of any current candidate for public office or displaying or distributing any campaign literature in support of such a candidate. No other kinds of posters or literature are subject to this prohibition, nor are any other types of commercial establishments in the state subject to similar prohibitions. Is this law constitutional?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}