{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"766ad163-f185-56af-a11c-8fa5ca2d60ca","task_key":"all--test--766ad163-f185-56af-a11c-8fa5ca2d60ca","task_revision_id":"1","upstream_id":"","short_description":"On February 15, a company that manufactures metal sidings for home exteriors…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the aluminum sheets were nonconforming goods.', 'Yes, because the company did not notify the builder that the 1/2-inch sheets were for accommodation only.', 'No, because the builder waived its right to reject the nonconforming goods by not returning them promptly to the company.', \\\"No, because the company could accept the builder's offer by prompt shipment of either conforming or nonconforming goods.\\\"]\",\"question\":\"On February 15, a company that manufactures metal sidings for home exteriors received the following order from a builder: \\\"Please ship 300 sheets of 1/4-inch refabricated aluminum siding. Delivery by April 1. \\\"On March 8, the company shipped 300 sheets of 1/2-inch refabricated aluminum siding, which were received by the builder on March 10. The following day, the builder sent the following fax to the company: \\\"Be advised that your shipment is rejected. Order stipulated 1/4-inch sheets. \\\" This fax was received by the company, but the builder did not ship the nonconforming aluminum sheets back to the company. Did the builder properly reject the shipment delivered on March 10?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}