{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0927d0de-7926-5712-ae66-5277d94e89c1","task_key":"default--test--1557","task_revision_id":"1","upstream_id":"1557","short_description":"A city has adopted the following ordinance which provides:\"Section 1: It shall…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['the section does not fall within the area of compelling state interest.', 'the section is content based.', 'the section permits the police chief to exercise unbridled discretion in approving or disapproving permits.', 'the section imposes undue financial burden on applicants.', 'the section does not specify a clear process for appeal against permit denial.', 'the section discriminates against certain religious beliefs.', 'the section violates the right to peaceful assembly.', 'the section does not provide specific criteria for denial of permits.']\",\"question\":\"A city has adopted the following ordinance which provides:\\\"Section 1: It shall be unlawful for any person, group, or organization to hold a meeting of 50 persons or more in any city park without first securing a city permit;Section 2: The application shall specify the day and hours for which the permit is sought. The fee shall be $10 per hour, with a maximum fee of $50;Section 3: Permits shall be issued on a first come basis; provided that the chief of police shall deny any application if, after hearing the applicant, it is his considered judgment that (a) the meeting would create serious traffic congestion, or (b) interfere with public enjoyment of the park, or (c) speakers at the meeting would advocate the commission of crime. \\\"A religious fundamentalist who believes that other religions should be violently resisted, planned to hold a protest demonstration against other religions at a city park on July 25. Although his previous anti-religion protest rallies attracted fewer than 25 demonstrators, he decided to apply for a permit pursuant to the city's ordinance. After meeting with the fundamentalist, the chief of police denied his permit application because he believed that the demonstration would incite the protestors and threaten imminent violence or serious disorder. On July 25, the fundamentalist and his fellow demonstrators staged their protest at a city park. The rally attracted only about 20 protestors and was conducted peacefully. As he was making his final remarks to the gathering, the city police arrived at the park and arrested him and his fellow demonstrators charging them with violating the city ordinance. If the fundamentalist now brings suit to challenge the constitutionality of Section 3 of the city ordinance, his best argument would be that\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}