{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d3c3af68-ff06-51a8-a4ca-c1d9f23f1c4a","task_key":"default--test--1616","task_revision_id":"1","upstream_id":"1616","short_description":"A buyer, located on the west coast contacted a seller, located on the east…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['succeed, because the accident happened before the goods reached the buyer.', 'succeed, because the risk of loss was on the seller.', 'not succeed, because the accident was an unforeseen event.', 'not succeed, because of frustration of purpose.', \\\"succeed, because the carrier was the seller's agent.\\\", 'succeed, because the seller has a responsibility to ensure safe delivery of goods.', 'succeed, because the seller should have insured the goods.', 'not succeed, because the risk of loss was on the buyer.']\",\"question\":\"A buyer, located on the west coast contacted a seller, located on the east coast, about purchasing flanges that the buyer used in their manufacturing process. The parties entered into a contract whereby the seller agreed to deliver to the buyer 1,000 flanges for $10,000. The contract stipulated that it was FOB at the seller's place of business. The seller then delivered the flanges to a delivery company, which was to transport them to the buyer. While en route, the vehicle that was transporting the flanges was involved in an accident and the flanges were destroyed. When the buyer received notice of the accident, the buyer immediately contacted the seller and demanded that it ship replacement flanges. The seller refused to do so. In an action by the buyer against the seller for breach of contract, the buyer will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}