{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7283ffd7-8ba2-583e-966e-c9e79e637779","task_key":"all--test--7283ffd7-8ba2-583e-966e-c9e79e637779","task_revision_id":"1","upstream_id":"","short_description":"An accountant had an office on the eighth floor of a downtown building. The…","config":"all","split":"test","body":"{\"choices\":\"[\\\"prevail, because the accountant is liable for its supplier's negligence.\\\", 'prevail, because the accountant instructed the designer to install plate glass rather than the more durable tempered glass.', 'not prevail, because the accountant could not reasonably have been expected to discover the defect.', 'not prevail, because the proper defendant is the manufacturer or the designer, not the accountant.']\",\"question\":\"An accountant had an office on the eighth floor of a downtown building. The accountant wanted to have renovations done to the entrance to the office. He hired an interior designer to do the work. The designer told the accountant that he had several options for the type of glass to be used in the door to the office. He could use half-inch thick plate glass, which would look nice and be relatively inexpensive. On the other hand, he could use tempered glass, which would cost twice as much but would be five times stronger than plate glass, more difficult to scratch, and capable of withstanding a blow from a hammer. The designer explained that the great majority of residential and small business establishments use plate glass for such doors. The accountant chose to go with the plate glass. The accountant ordered a plate glass door from a manufacturer and had the designer install the door when it arrived. A few months later, the door shattered when one of the accountant's clients pulled on the handle to open it. The glass cut the client's hand. If the client asserts a claim against the accountant based on negligence and establishes negligent manufacture of the glass door by the manufacturer that was not discoverable through reasonable inspection, the plaintiff will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}