{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d967cb15-b8b0-5e60-9b37-5651c87291d5","task_key":"default--test--1951","task_revision_id":"1","upstream_id":"1951","short_description":"Fernandez is planning to attend an auction of the assets of Cross & Black, one…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Such a sale is without reserve unless the goods are explicitly put up with reserve.', 'A bidder may retract his bid at any time until the falling of the hammer.', 'The retraction of a bid by a bidder revives the previous bid.', 'If the auction is without reserve, the auctioneer can withdraw the article at any time prior to the fall of the hammer.']\",\"question\":\"Fernandez is planning to attend an auction of the assets of Cross & Black, one of his major competitors who is liquidating. In the conduct of the auction, which of the following rules applies?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}