{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"170fddbd-058b-58cf-a67a-62bfe80b326f","task_key":"default--test--1431","task_revision_id":"1","upstream_id":"1431","short_description":"A contractor agreed to remodel a homeowner's garage for $5,000. Just before the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the oral agreement contradicts the written contract.', 'Yes, because the homeowner brought up the time limit before signing the contract.', 'Yes, because the homeowner can provide evidence of the oral agreement.', 'Yes, because the time limit is an additional term that does not contradict the partially integrated written contract.', 'No, because the oral agreement is considered hearsay.', 'No, because the contract is a fully integrated written contract.', 'Yes, because the court can consider oral agreements in absence of a merger clause.', 'No, because the court must ascertain the meaning of the agreement from the terms of the written contract.', 'Yes, because the contract is ambiguous.', \\\"No, because the oral agreement was merely part of the parties' negotiations.\\\"]\",\"question\":\"A contractor agreed to remodel a homeowner's garage for $5,000. Just before the parties signed the one-page written contract, the homeowner called to the contractor's attention the fact that the contract did not specify a time of completion. The parties orally agreed but did not specify in the contract that the contractor would complete the work in 60 days, and then they both signed the contract. The contract did not contain a merger clause. The contractor failed to finish the work in 60 days. The homeowner has sued the contractor for breach of contract. Is the court likely to admit evidence concerning the parties' oral agreement that the work would be completed in 60 days?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}