{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6f1f1d1e-bd49-52f1-b82a-a321c9f55420","task_key":"all--test--6f1f1d1e-bd49-52f1-b82a-a321c9f55420","task_revision_id":"1","upstream_id":"","short_description":"A man was driving his new car along a dark road with the car's high-beam…","config":"all","split":"test","body":"{\"choices\":\"['Only the manufacturer of the car and the retailer of the car.', 'Only the manufacturer of the car.', 'Only the manufacturer of the headlight controls and the manufacturer of the car.', 'All three defendants.']\",\"question\":\"A man was driving his new car along a dark road with the car's high-beam headlights on to illuminate the road. When he saw the headlights of another car appear in the distance, he reached to turn the high beams off. Instead of turning from high-beam to low-beam, the headlights on the car turned off completely. The man tried repeatedly to turn the lights on again but could not do so. He collided with the other car and suffered injuries. The man has brought an action against the manufacturer of the headlight controls in his car, the manufacturer of his car, and the retailer who sold him the car. If the man can establish that a defect in the controls caused the accident, from whom can he recover?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}