{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be10a2af-484a-54df-b03e-8a2f41d4805e","task_key":"default--test--1403","task_revision_id":"1","upstream_id":"1403","short_description":"A homeowner and a contractor entered into a contract under which the homeowner…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, because the homeowner's payment of $35,000 cannot furnish consideration for the contractor's relinquishment of a claim for $50,000.\\\", 'No, because the contractor agreed to the settlement and accepted the $35,000 payment.', 'No, because the homeowner has the right to dispute the quality of work and adjust payment accordingly.', 'Yes, because the contractor has provided services worth $50,000 as initially agreed.', 'No, because the reasonable value of the work was only $35,000.', 'No, because the homeowner honestly disputed the amount he owed the contractor.', 'Yes, because the homeowner has breached the contract by not paying the full amount agreed upon.', \\\"Yes, because the contractor reasonably relied on the homeowner's contractual promise to pay the full $50,000, and that promise should be enforced to avoid injustice.\\\", \\\"Yes, because the homeowner's belief of defects and departures from the plan were unfounded.\\\", 'No, because the contractor willingly accepted the new offer, thus forming a new agreement.']\",\"question\":\"A homeowner and a contractor entered into a contract under which the homeowner agreed to pay the contractor $50,000 for remodeling the homeowner's basement according to a set of plans. After the work was completed, the homeowner honestly believed that there were defects in the contractor's work as well as departures from the plans. In fact, the contractor had fully performed. The homeowner offered to pay the contractor $35,000 in full settlement in exchange for the contractor's promise to surrender his entire claim. The contractor accepted the homeowner's offer, and the homeowner paid the contractor $35,000. The reasonable value of the work was $35,000. Is the contractor likely to succeed in an action challenging the validity of the settlement agreement?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}