{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4cd627c8-c8a1-53e3-b74b-f248aee9724d","task_key":"all--test--4cd627c8-c8a1-53e3-b74b-f248aee9724d","task_revision_id":"1","upstream_id":"","short_description":"An owner of a tract of land in a foothill community hired a general contractor…","config":"all","split":"test","body":"{\"choices\":\"['order the architect to execute the certificate.', 'require the contractor to proceed in equity for reformation to eliminate the clause.', 'award the contractor damages against the architect.', 'excuse the condition and require the owner to pay damages to the contractor.']\",\"question\":\"An owner of a tract of land in a foothill community hired a general contractor to construct a small summer cottage on the tract of land. The parties entered into a valid, written construction contract that contained a provision that called for an architect's certificate of completion. After completion of the cottage, the owner hired an architect to inspect the contractor's work and issue the certificate. The architect refused in bad faith to execute the certificate. The contractor then sued the owner for the entire contract amount. The owner argued that the lack of issuance of the certificate excused her from further performance under the contract. Upon hearing this argument, most courts will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}