{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1a5d233f-7a6c-57ad-9c27-7820d3f5ba23","task_key":"default--test--1967","task_revision_id":"1","upstream_id":"1967","short_description":"An owner of a large, three-story office building planned to renovate the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"$75,000 only, due to the owner's lack of involvement in the subcontractor's bid.\\\", \\\"$85,000, to cover the additional cost and the inconvenience caused by the first electrician's withdrawal.\\\", 'only in quantum meruit, because the contractor performed the renovation work despite the disagreement over costs.', \\\"$75,000 only, because the contractor failed to secure the electrician's commitment.\\\", '$75,000 only, because that was the contract price.', '$80,000, because the owner should bear the cost of unexpected expenses.', \\\"$80,000, because the contractor reasonably relied to his detriment on the electrician's bid in formulating his job estimate.\\\", '$80,000, because the owner was informed about the increased cost before work began.', 'only in quantum meruit, because of the doctrifle of commercial frustration.', 'only in quantum meruit, because by demanding $80,000 the contractor, in effect, repudiated his contract with the owner.']\",\"question\":\"An owner of a large, three-story office building planned to renovate the building and requested several contractors to submit bids to perform the work. One of the contractors in turn sought bids from several subcontractors to determine the overall cost of the job. An electrician submitted a bid to do the electrical work to the contractor, which the contractor used in computing the bid he sent to the owner. The contractor notified the electrician to that effect. Two weeks later, the contractor submitted his bid to the owner, in which he offered to perform the renovation work for $75,000. The next day, the electrician notified the contractor that he was unable to perform the electrical work on the owner's project. Two days later, the owner sent the contractor a signed confirmation letter wherein she stated, \\\"I hereby accept your offer to perform the renovation work for the quoted cost of $75,000. \\\"Thereafter, the contractor hired another electrician to complete the electrical work at a cost of $5,000 above the first electrician's bid. The contractor explained the situation to the owner and informed her that the overall cost of the job would have to be increased to $80,000. The owner responded that she would hold the contractor to his original bid of $75,000 and would not be responsible for any additional costs. The contractor then performed the renovation work, but the owner has not yet paid him anything. In an action by the contractor against the owner for the services rendered, the contractor will probably be able to recover\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}