{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93df56c1-8dba-5211-9efc-e613f9443373","task_key":"all--test--93df56c1-8dba-5211-9efc-e613f9443373","task_revision_id":"1","upstream_id":"","short_description":"A national distributor of windows selected a retailer to sell its windows in a…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, because the retailer's forwarding of orders to the distributor did not give rise to an obligation on the distributor's part to fill the orders.\\\", \\\"No, because the retailer's practice of overcharging customers excused the distributor's refusal to fill the order.\\\", \\\"Yes, because the distributor's claim regarding overcharging customers is independent of its obligation to fill the retailer's order.\\\", \\\"Yes, because the parties' course of dealing binds the distributor to fill the retailer's order.\\\"]\",\"question\":\"A national distributor of windows selected a retailer to sell its windows in a specified geographic area. The parties negotiated a written distribution agreement, which stated that any order for windows placed by the retailer would be binding on the distributor \\\"only when expressly accepted by the distributor.\\\" For the next two years, the retailer forwarded orders to the distributor, and the distributor always filled the orders. In the third year, the distributor accused the retailer of overcharging customers to install the distributor's windows. The retailer responded that the distributor had no control over the retailer's installation prices. When the distributor received the retailer's next order for windows, it refused to fill the order. If the retailer sues the distributor for breach of contract, will it be likely to prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}