{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"56bcff1f-a203-5ef6-b448-d94f54be9427","task_key":"all--test--56bcff1f-a203-5ef6-b448-d94f54be9427","task_revision_id":"1","upstream_id":"","short_description":"To preserve the appearance and accessibility of its capitol building, a state…","config":"all","split":"test","body":"{\"choices\":\"['No, because even though the sidewalk at issue is not a public forum, the prohibition against the display of signs is more restrictive than needed to serve a legitimate government interest. government interest.', 'No, because the sidewalk at issue is a public forum, and the prohibition against the display of signs is not narrowly tailored to serve a substantial government interest.', 'Yes, because even though the sidewalk at issue is a public forum, the prohibition against the display of signs is necessary to serve a compelling public interest.', 'Yes, because the sidewalk at issue is not a public forum, and the prohibition against the display of signs is reasonable.']\",\"question\":\"To preserve the appearance and accessibility of its capitol building, a state enacted a law prohibiting \\\"the display of any sign on any portion of the public sidewalk surrounding\\\" the building. A group of five demonstrators who wanted to protest inadequate state funding for children's services applied for a permit to march single file on the sidewalk surrounding the capitol building. Each demonstrator planned to carry a two-foot-square sign that would read, \\\"Our lawmakers do not care about our children.\\\" The group's permit application was denied pursuant to the state law, and the group has filed an action challenging the law's constitutionality. Should the court uphold the law's constitutionality?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}