{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ee72524-fcb7-5f1e-a3c6-2440880fe579","task_key":"all--test--8ee72524-fcb7-5f1e-a3c6-2440880fe579","task_revision_id":"1","upstream_id":"","short_description":"Fernandez is planning to attend an auction of the assets of Cross & Black, one…","config":"all","split":"test","body":"{\"choices\":\"['Such a sale is without reserve unless the goods are explicitly put up with reserve.', 'A bidder may retract his bid at any time until the falling of the hammer.', 'The retraction of a bid by a bidder revives the previous bid.', 'If the auction is without reserve, the auctioneer can withdraw the article at any time prior to the fall of the hammer.']\",\"question\":\"Fernandez is planning to attend an auction of the assets of Cross & Black, one of his major competitors who is liquidating. In the conduct of the auction, which of the following rules applies?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}