{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f21f4b04-3e9d-506b-b2d6-e22488d490d2","task_key":"all--test--f21f4b04-3e9d-506b-b2d6-e22488d490d2","task_revision_id":"1","upstream_id":"","short_description":"A person purchased a new car from a local auto dealership. Over the next month,…","config":"all","split":"test","body":"{\"choices\":\"['Yes, if the brakes were defective when the dealership sold the car to the purchaser.', 'Yes, because the dealership is strictly liable for defective repairs on cars it has sold.', 'No, because the car had been driven more than 1,000 miles.', \\\"No, unless the dealership's employee was negligent in repairing the brakes.\\\"]\",\"question\":\"A person purchased a new car from a local auto dealership. Over the next month, the purchaser drove the car a little more than 1,000 miles. The purchaser noticed that the car made a squeaking noise when the brakes were applied, so the purchaser took the car back to the dealership to have the mechanic there take a look at it. The mechanic worked on the car and then told the purchaser that the problem had been fixed. A few days later, the brakes failed, causing the purchaser to crash into a tree. The purchaser suffered a serious head injury in the crash. If the purchaser asserts a claim against the car dealership for damages for his injuries, will the purchaser prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}