{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"95da15d1-9325-5688-98a5-496d7b12bcef","task_key":"all--test--95da15d1-9325-5688-98a5-496d7b12bcef","task_revision_id":"1","upstream_id":"","short_description":"A doctor parked her car in a public parking lot in a downtown area. A…","config":"all","split":"test","body":"{\"choices\":\"['No, because the doctor had auto insurance, so she suffered no loss.', 'No, because the manufacturer was unaware of the defect in the crane.', 'Yes, because the manufacturer sold a product with a dangerous defect.', 'Yes, because the manufacturer failed to inspect the crane adequately before selling it.']\",\"question\":\"A doctor parked her car in a public parking lot in a downtown area. A construction company was using a large crane to build a new office building next to the parking lot. When the crane was manufactured, some of the welding of its parts was done improperly. No one had ever noticed this manufacturing defect in the crane. The defect caused the crane to break and topple over. Part of the crane fell on the doctor's car. The doctor's car was completely destroyed. The car was valued at $35,000. The doctor had auto insurance, and the insurance company promptly paid the doctor $35,000 under the insurance policy. If the doctor brings suit against the manufacturer of the crane, will the doctor prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}