{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4bc7236-6a9e-57e4-9502-d2092cc9f5dc","task_key":"all--test--c4bc7236-6a9e-57e4-9502-d2092cc9f5dc","task_revision_id":"1","upstream_id":"","short_description":"A farmer purchased a new pickup truck from a local automobile dealership. A few…","config":"all","split":"test","body":"{\"choices\":\"['No, because the pedestrian was not a purchaser or user of the truck.', 'No, if the farmer was negligent in being unable to steer around and avoid hitting the pedestrian.', 'Yes, because the auto dealer placed a defective car into the stream of commerce.', 'Yes, if the defect could have been discovered through the exercise of reasonable care by the auto dealer.']\",\"question\":\"A farmer purchased a new pickup truck from a local automobile dealership. A few weeks later, the farmer was using the truck to haul a load of supplies to his farm. As he drove down a small hill toward an intersection with a stop sign, he applied the brakes, but they failed to work, and the truck did not slow down. The farmer could not stop for the stop sign. He saw a pedestrian crossing the street at the intersection and tried to steer around him, but was unable to do so. The pickup truck struck the pedestrian and seriously injured him. If the injured pedestrian asserts a negligence claim against the auto dealership that sold the truck to the farmer, and if it is conclusively proven that the brake failure resulted from a manufacturing defect in the car, will the pedestrian prevail?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}