{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6d1f63a5-fd70-52ea-84ef-20fd0216e685","task_key":"default--test--1228","task_revision_id":"1","upstream_id":"1228","short_description":"A homeowner contracted in writing with a kitchen contractor to renovate her…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the homeowner cannot base the payment on personal satisfaction.', 'Yes, because the homeowner breached the covenant of good faith and fair dealing by rejecting the cabinets without justification.', 'No, because the contractor breached his duty of good faith and fair dealing by supplying unsatisfactory materials.', \\\"No, because the contractor should have ensured the homeowner's satisfaction before seeking payment.\\\", 'Yes, because the contractor has the right to sue for the balance of the contract price.', 'Yes, because the homeowner was the first party to breach the contract.', \\\"No, because a condition to the homeowner's obligation to pay was not satisfied.\\\", 'No, because the homeowner was not satisfied with the finish on the cabinets.', 'Yes, because the contractor fulfilled his part of the contract by completing the renovation.', 'No, because the homeowner has a right to withhold payment if not completely satisfied.']\",\"question\":\"A homeowner contracted in writing with a kitchen contractor to renovate her kitchen for $25,000, \\\"subject to the homeowner's complete personal satisfaction.\\\" The contractor replaced the cabinets, flooring, and countertops and then sought payment from the homeowner. The homeowner paid the contractor only $20,000, truthfully saying that she did not like the finish on the cabinets and was therefore not satisfied. If the contractor sues the homeowner for the balance of the contract price, will the contractor be likely to prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}