{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c1ae21b-4db5-574c-9faf-2f82199f6f4f","task_key":"all--test--7c1ae21b-4db5-574c-9faf-2f82199f6f4f","task_revision_id":"1","upstream_id":"","short_description":"A homeowner contracted in writing with a kitchen contractor to renovate her…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, because a condition to the homeowner's obligation to pay was not satisfied.\\\", 'No, because the contractor breached his duty of good faith and fair dealing by supplying unsatisfactory materials.', 'Yes, because the homeowner breached the covenant of good faith and fair dealing by rejecting the cabinets without justification.', 'Yes, because the homeowner was the first party to breach the contract.']\",\"question\":\"A homeowner contracted in writing with a kitchen contractor to renovate her kitchen for $25,000, \\\"subject to the homeowner's complete personal satisfaction.\\\" The contractor replaced the cabinets, flooring, and countertops and then sought payment from the homeowner. The homeowner paid the contractor only $20,000, truthfully saying that she did not like the finish on the cabinets and was therefore not satisfied. If the contractor sues the homeowner for the balance of the contract price, will the contractor be likely to prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}