{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e45dbb89-4635-5629-a75c-488303f81ca8","task_key":"all--test--e45dbb89-4635-5629-a75c-488303f81ca8","task_revision_id":"1","upstream_id":"","short_description":"A supermarket had just reopened after a six-month renovation period. The…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the supermarket failed to take adequate precaution to ensure the safety of its patrons against such an unreasonable risk of harm.', 'prevail, because the supermarket is vicariously liable for the negligence of its independent contractor in the performance of a non-delegable duty.', 'not prevail, because the injury was the fault of the painting company.', 'not prevail, because the supermarket posted signs warning customers of the risk of harm.']\",\"question\":\"A supermarket had just reopened after a six-month renovation period. The renovations included the repair, replastering, and repainting of the entire ceiling and walls by the painting company. The day following the reopening of the supermarket, the store's manager noticed small fragments of plaster on the floor, which appeared to have fallen from a part of the ceiling about 10-square-feet in area. The manager immediately posted signs in that area that read: \\\"Caution: Falling Plaster. \\\" In addition, the manager promptly called the painting company and requested a repairman to attend to the problem. That afternoon, a shopper was shopping in the supermarket and wanted to purchase some canned goods in the posted area. Moving quickly, the shopper reached for the desired items. At that very moment, a section of the ceiling fell and struck the shopper, injuring her very seriously. If the shopper asserts a claim against the supermarket, she probably will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}