{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"72ee5150-47f1-59bf-ba14-3306341832b4","task_key":"all--test--72ee5150-47f1-59bf-ba14-3306341832b4","task_revision_id":"1","upstream_id":"","short_description":"A graduate of law school received notice that she had successfully passed the…","config":"all","split":"test","body":"{\"choices\":\"[\\\"prevail, because the graduate's conduct was extreme and outrageous.\\\", 'prevail, because an offensive touching resulted.', \\\"not prevail, because the graduate's actions were involuntary.\\\", 'not prevail, because the graduate was unaware what she was drinking.']\",\"question\":\"A graduate of law school received notice that she had successfully passed the bar exam. To celebrate passing, the graduate went out with a few friends to a popular campus bar. The graduate's friend ordered a round of Hula Girls for everyone. A Hula Girl is an extremely potent alcoholic drink consisting. of 2 ounces of dry gin, 1 ounce of French vermouth, and 1 teaspoon of grenadine. After the drink is mixed, it is then served in a glass with an inverted lemon peel, which is cut at the ends to represent a grass skirt. Although the graduate had never heard of a Hula Girl, she was in a festive mood and drank it, anyway. A few minutes later, the graduate became very dizzy and attempted to stand up from the barstool. As she tried to walk to the restroom, she became nauseated and vomited over a customer sitting at a nearby table. The customer was embarrassed and greatly humiliated by the incident. If the customer asserts a claim against the graduate, the plaintiff will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}