{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ffcf3085-7352-5ae7-badd-c45553879536","task_key":"all--test--ffcf3085-7352-5ae7-badd-c45553879536","task_revision_id":"1","upstream_id":"","short_description":"A woman owned four beach houses. Each house had star-shaped windows. As the…","config":"all","split":"test","body":"{\"choices\":\"[\\\"succeed, because the carpenter's loss of the inventory would not excuse his duty of performance.\\\", 'succeed, because the beach houses would suffer severe damage in the event a hurricane struck without the storm windows having been installed.', \\\"not succeed, because the carpenter's performance would be excused by the unforeseeable act of God.\\\", \\\"not succeed, because the carpenter's loss of inventory would render his performance impossible.\\\"]\",\"question\":\"A woman owned four beach houses. Each house had star-shaped windows. As the hurricane season was approaching, she hired a carpenter to build storm windows for the star-shaped and deliver them to the four houses. Because of the unique shape of the windows, the carpenter had to do all work by hand. The carpenter and woman signed a contract under which the carpenter promised to build and deliver the storm windows to all four beach houses within 30 days, and the woman promised to pay $8,000 ($2,000 per beach house) within one week of completion of the job. Two weeks after the contract was signed, a fire destroyed the carpenter's workshop, where he was storing all the completed storm windows before delivery. The carpenter then sent a letter to the woman that read:\\\"The fire destroyed my equipment and inventory €” including the storm windows, which had been completed and which were awaiting delivery. I am dead broke and cannot complete my obligations under the contract. \\\" The woman then telephoned the carpenter and told him, \\\"Unless you fulfill your contractual obligations, I will sue you!\\\"In an action for specific performance to compel the carpenter to build and deliver the storm windows, the woman will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}