{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"828f2c0c-c1cd-5e0c-842e-e34388ca2df4","task_key":"all--test--828f2c0c-c1cd-5e0c-842e-e34388ca2df4","task_revision_id":"1","upstream_id":"","short_description":"A customer went into a bar to have a drink. The bar was crowded because a…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because it was reasonably foreseeable that fights would occur in a crowd of people who were drinking a lot of alcohol.', 'prevail, because the person who punched the customer was employed as a bouncer.', 'not prevail, because the owner did not touch the customer.', 'not prevail, because the owner should not be responsible for the intentional acts of the employee.']\",\"question\":\"A customer went into a bar to have a drink. The bar was crowded because a championship boxing match was being shown on the television sets in the bar, and many people had gone to the bar to watch it. During the second round of the boxing match, someone bumped into the customer, causing him to spill a drink on his pants. \\\"Hey, watch what you're doing, jerk!\\\" the customer said. The person who had bumped the customer was an employee of the bar hired as a bouncer to deal with unruly customers. Angered by the customer's statement, the employee immediately turned around and punched the customer in the mouth. If the customer asserts a battery claim against the owner of the bar, the customer will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}