{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"556c75e2-d449-5f2a-99a3-c9b9c32c5f51","task_key":"default--test--1935","task_revision_id":"1","upstream_id":"1935","short_description":"A man placed the following advertisement in his local newspaper:\"Public Auction…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"prevail, because the auctioneer asked for other bids after the woman's opening bid, validating her bid.\\\", 'prevail, because whether or not the auction is \\\"without reserve,\\\" goods cannot be withdrawn after the auctioneer calls for bids.', 'not prevail, because at an auction \\\"without reserve,\\\" the auctioneer may withdraw goods until he announces completion of the sale.', \\\"not prevail, because the auctioneer did not formally accept the woman's bid before withdrawing the piano.\\\", 'not prevail, because at an auction \\\"without reserve,\\\" the auctioneer invites offers, which he may accept or reject.', 'prevail, because the auctioneer stated that the piano is worth at least $5,000, implying a minimum bid.', 'not prevail, because the auctioneer has the discretion to withdraw any item from an auction.', 'prevail, because goods put up at an auction \\\"without reserve\\\" may not be withdrawn.', 'prevail, because the auction was advertised as \\\"without reserve,\\\" meaning all items must be sold to the highest bidder.', 'not prevail, because the auctioneer has the right to reject any bid he considers too low.']\",\"question\":\"A man placed the following advertisement in his local newspaper:\\\"Public Auction Sale . . . without reserve December 7, 10:00 a. m. 110 Walnut St. , City, StateEntire furnishings of home must go: antiques, piano, pool table, appliances, tables, stereo system, etc. \\\"On the morning of December 7, a group of approximately 20 people showed up at the man's home. The first item the man put up for bid was an antique Baldwin grand piano. The man announced that the bidding would be \\\"without reserve\\\" and said, \\\"What's the opening bid on this beautiful Baldwin grand piano?\\\" A woman opened the bidding with $100. The man then commented, \\\"This piano is worth at least $5,000. What other bids am I offered?\\\" When the man did not receive any other bids, he informed the woman that he would not accept $100 and was removing the piano from the sale. If the woman asserts an action against the man for breach of contract, the woman will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}