{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"49c9aebf-4f49-52fd-a7a3-eeb966342231","task_key":"all--test--49c9aebf-4f49-52fd-a7a3-eeb966342231","task_revision_id":"1","upstream_id":"","short_description":"A homeowner and a contractor entered into a contract under which the homeowner…","config":"all","split":"test","body":"{\"choices\":\"['No, because the homeowner honestly disputed the amount he owed the contractor.', 'No, because the reasonable value of the work was only $35,000.', \\\"Yes, because the contractor reasonably relied on the homeowner's contractual promise to pay the full $50,000, and that promise should be enforced to avoid injustice.\\\", \\\"Yes, because the homeowner's payment of $35,000 cannot furnish consideration for the contractor's relinquishment of a claim for $50,000.\\\"]\",\"question\":\"A homeowner and a contractor entered into a contract under which the homeowner agreed to pay the contractor $50,000 for remodeling the homeowner's basement according to a set of plans. After the work was completed, the homeowner honestly believed that there were defects in the contractor's work as well as departures from the plans. In fact, the contractor had fully performed. The homeowner offered to pay the contractor $35,000 in full settlement in exchange for the contractor's promise to surrender his entire claim. The contractor accepted the homeowner's offer, and the homeowner paid the contractor $35,000. The reasonable value of the work was $35,000. Is the contractor likely to succeed in an action challenging the validity of the settlement agreement?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}