{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6872436f-95aa-525a-b376-4fd5b49b9bd5","task_key":"all--test--6872436f-95aa-525a-b376-4fd5b49b9bd5","task_revision_id":"1","upstream_id":"","short_description":"The owner of a test prep company sent an advertisement regarding the company's…","config":"all","split":"test","body":"{\"choices\":\"['The owner, because such advertising is protected by the First Amendment under the commercial speech doctrine.', 'The owner, because there is a constitutional right of advertising under the First and Fourteenth Amendments.', 'The editor and newspaper, because Congress is empowered to prohibit untruthful advertising, even where it urges the purchase of a legal, validly offered item.', 'The editor and newspaper, because there is no constitutional right of advertising under the First and Fourteenth Amendments.']\",\"question\":\"The owner of a test prep company sent an advertisement regarding the company's review courses to a local newspaper. In an accompanying letter, the owner instructed the newspaper to publish the ad in an upcoming edition. The ad was received by the newspaper's advertising editor. The next day the editor phoned the owner and told her that he would not permit the newspaper to run the ad. When the owner asked for an explanation, the editor replied, \\\"My daughter took your review course and scored low on her entrance exams to college and didn't get into the college of her choice. Because of your instructors, she's now working at the mall. That's why I'm prohibiting the newspaper from publishing your ad. \\\" The editor then forwarded a letter to the owner reiterating his newspaper's refusal to have the ad published. In an appropriate action, the owner brings suit against the editor and the newspaper seeking an order that would require the newspaper to publish the advertisement. In such action, who will prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}