{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"646c104f-4b54-5c6d-a66b-3ac528185bd3","task_key":"all--test--646c104f-4b54-5c6d-a66b-3ac528185bd3","task_revision_id":"1","upstream_id":"","short_description":"A company manufactures a popular dandruff shampoo. On the shampoo's box cover is…","config":"all","split":"test","body":"{\"choices\":\"['win, because she suffered injury from her use of the product.', 'win, because the manufacturer was aware that a small percentage of the population would suffer an allergic reaction to the shampoo.', \\\"lose, because she didn't read or speak English.\\\", 'lose, because she assumed the risk by not making the \\\"patch test. \\\"']\",\"question\":\"A company manufactures a popular dandruff shampoo. On the shampoo's box cover is a warning label advising consumers to make a \\\"patch test\\\" before applying. The label states that a \\\"patch test\\\" is necessary because a small percentage of the population may be allergic to the chemicals contained in the product and susceptible to suffering scalp irritation. A Russian immigrant recently moved to the United States from Moscow. The immigrant cannot read or speak English. One day, the immigrant purchased a bottle of the shampoo from a drug store. She looked at the label warning but did not understand it. After throwing away the box cover, she used the shampoo without making a \\\"patch test. \\\" Minutes later, she began experiencing an allergic reaction and scalp irritation. This was followed by hair loss attributed to the shampoo. If the immigrant brings suit against the company for strict products liability, she will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}