{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f719c8c1-40f5-512d-a05a-f74ee42b5f6b","task_key":"default--test--1608","task_revision_id":"1","upstream_id":"1608","short_description":"A contractor and home owner were bargaining on the price for the construction of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The home owner tells the contractor they have decided to build the house themselves.', 'The contractor informs the home owner that the cost of materials has increased, therefore the offer needs to be renegotiated.', 'The home owner asks the contractor if they would be willing to build the house for $95,000.', 'The home owner asks the contractor if they could complete the job within six months.', \\\"The contractor dies before the home owner accepts but the contractor's son intends to continue the business.\\\", 'The home owner tells the contractor they have found another contractor who will do the job for $90,000.', 'The home owner states \\\"I accept your offer but the price is to be $97,000.\\\"', 'The contractor contacts the home owner and states that the offer is withdrawn.', 'The contractor increases the offer to $105,000 before the homeowner can accept.', \\\"The contractor's business is declared bankrupt before the home owner can accept.\\\"]\",\"question\":\"A contractor and home owner were bargaining on the price for the construction of a new home. The contractor made a number of offers for construction to the home owner including one for $100,000. Which of the following communications would not terminate the offer so that a subsequent acceptance could be effective\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}