{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"52320964-60f3-55f8-a5a7-170cd98db18f","task_key":"all--test--52320964-60f3-55f8-a5a7-170cd98db18f","task_revision_id":"1","upstream_id":"","short_description":"A woman hired a builder to build a house according to certain plans and…","config":"all","split":"test","body":"{\"choices\":\"[\\\"Yes, because by operation of law the woman is an equitable assignee of the builder's claim against the owner of the lumber yard for breach of contract.\\\", \\\"Yes, but only if the builder's contract with the owner of the lumber yard was not discharged by the fire.\\\", 'No, because privity of contract does not exist between the woman and the owner of the lumber yard.', 'No, because the woman is only an incidental beneficiary of the contract between the builder and the owner of the lumber yard.']\",\"question\":\"A woman hired a builder to build a house according to certain plans and specifications prepared by the woman's architect. The agreed upon price was $250,000, with construction to be completed within four months. Two weeks after the building contract was formed, the builder contacted a lumber yard to purchase wood necessary for the construction of the house. The builder and the owner of the lumber yard entered into a valid written agreement whereby the lumber yard was to supply the necessary lumber in exchange for $10,000, payable by the builder within 30 days. One week later, a fire destroyed a good portion of the lumber yard's supply of lumber. As a result, the lumber yard refused to supply lumber to the builder. The builder was unable to find another supplier of lumber and therefore notified the woman that he would be unable to complete her building on time. If the woman sues the owner of the lumber yard for breach of contract, will she prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}