{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d94c7fe8-bb92-574f-8446-c8d635d01dfd","task_key":"all--test--d94c7fe8-bb92-574f-8446-c8d635d01dfd","task_revision_id":"1","upstream_id":"","short_description":"A man placed the following advertisement in his local newspaper:\"Public Auction…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because goods put up at an auction \\\"without reserve\\\" may not be withdrawn.', 'prevail, because whether or not the auction is \\\"without reserve,\\\" goods cannot be withdrawn after the auctioneer calls for bids.', 'not prevail, because at an auction \\\"without reserve,\\\" the auctioneer may withdraw goods until he announces completion of the sale.', 'not prevail, because at an auction \\\"without reserve,\\\" the auctioneer invites offers, which he may accept or reject.']\",\"question\":\"A man placed the following advertisement in his local newspaper:\\\"Public Auction Sale . . . without reserve December 7, 10:00 a. m. 110 Walnut St. , City, StateEntire furnishings of home must go: antiques, piano, pool table, appliances, tables, stereo system, etc. \\\"On the morning of December 7, a group of approximately 20 people showed up at the man's home. The first item the man put up for bid was an antique Baldwin grand piano. The man announced that the bidding would be \\\"without reserve\\\" and said, \\\"What's the opening bid on this beautiful Baldwin grand piano?\\\" A woman opened the bidding with $100. The man then commented, \\\"This piano is worth at least $5,000. What other bids am I offered?\\\" When the man did not receive any other bids, he informed the woman that he would not accept $100 and was removing the piano from the sale. If the woman asserts an action against the man for breach of contract, the woman will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}