{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0e8af250-fa2e-5e64-aefd-bceb0a64cbaf","task_key":"default--test--1562","task_revision_id":"1","upstream_id":"1562","short_description":"The owner of a test prep company sent an advertisement regarding the company's…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The editor and newspaper, because there is no constitutional right of advertising under the First and Fourteenth Amendments.', 'The editor and newspaper, because they have the right to control the content that is published in their newspaper.', 'The editor and newspaper, because they have the right to refuse any advertisement they deem unsuitable.', 'The editor and newspaper, because Congress is empowered to prohibit untruthful advertising, even where it urges the purchase of a legal, validly offered item.', \\\"The owner, because the editor's personal bias should not affect the newspaper's business decisions.\\\", \\\"The owner, because the newspaper's refusal to publish the ad constitutes a breach of contract.\\\", 'The owner, because the refusal to publish the ad is a form of discrimination.', 'The owner, because such advertising is protected by the First Amendment under the commercial speech doctrine.', 'The editor and newspaper, because their decision is based on personal experience with the service.', 'The owner, because there is a constitutional right of advertising under the First and Fourteenth Amendments.']\",\"question\":\"The owner of a test prep company sent an advertisement regarding the company's review courses to a local newspaper. In an accompanying letter, the owner instructed the newspaper to publish the ad in an upcoming edition. The ad was received by the newspaper's advertising editor. The next day the editor phoned the owner and told her that he would not permit the newspaper to run the ad. When the owner asked for an explanation, the editor replied, \\\"My daughter took your review course and scored low on her entrance exams to college and didn't get into the college of her choice. Because of your instructors, she's now working at the mall. That's why I'm prohibiting the newspaper from publishing your ad. \\\" The editor then forwarded a letter to the owner reiterating his newspaper's refusal to have the ad published. In an appropriate action, the owner brings suit against the editor and the newspaper seeking an order that would require the newspaper to publish the advertisement. In such action, who will prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}