{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e1585c20-eb45-5386-abc2-582f661e92e1","task_key":"all--test--e1585c20-eb45-5386-abc2-582f661e92e1","task_revision_id":"1","upstream_id":"","short_description":"A city has undergone a massive redevelopment project aimed at remodeling and…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the ordinance violates his freedom of speech.', 'prevail, because such a time, place, and manner restriction on private property is discriminatory and overbroad.', 'not prevail, because although commercial speech is protected by the First Amendment, it is subject to greater regulation than other forms of protected speech.', 'not prevail, because the ordinance is rationally related to a legitimate state interest.']\",\"question\":\"A city has undergone a massive redevelopment project aimed at remodeling and beautifying the downtown area. Recently the city council passed an ordinance prohibiting the placement of any sign with dimensions larger than six feet on the exterior of any commercial building. Furthermore, signs within the guidelines of the ordinance could only relate to advertising the business of the property's occupant. The intended purpose of the ordinance was to advance the municipality's interests in traffic safety and aesthetics. An owner of an office building in the newly developed section of town, placed a 10-foot sign on the outside of his building endorsing his brother's political candidacy. The city council has ordered the owner to remove the sign from his building, but he has refused to take it down until after the election. If the owner challenges the city ordinance, the most likely result is that he will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}