{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e5b70cb0-2f31-5a04-a283-27606fa743e1","task_key":"default--test--1952","task_revision_id":"1","upstream_id":"1952","short_description":"Fernandez is planning to attend an auction of the assets of Cross & Black, one…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['A bidder can retract his bid even after the fall of the hammer.', 'If the auction is with reserve, the auctioneer cannot withdraw the article at any time.', 'The retraction of a bid by a bidder automatically ends the auction.', 'A bidder may retract his bid at any time until the falling of the hammer.', 'The auctioneer can increase the bidding price at any time during the auction.', 'If the auction is without reserve, the auctioneer can withdraw the article at any time prior to the fall of the hammer.', 'All bids are final and cannot be retracted under any circumstances.']\",\"question\":\"Fernandez is planning to attend an auction of the assets of Cross & Black, one of his major competitors who is liquidating. In the conduct of the auction, which of the following rules applies?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}