{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"48e14d77-6bde-576a-987c-ae0c1c4e978a","task_key":"all--test--48e14d77-6bde-576a-987c-ae0c1c4e978a","task_revision_id":"1","upstream_id":"","short_description":"A carpenter was driving home one night after a long day of work. He was driving…","config":"all","split":"test","body":"{\"choices\":\"['The bus driver had received three speeding tickets in the past.', \\\"The carpenter's car was in good condition, and it had passed a safety inspection just a week before the accident.\\\", 'The traffic signal at the intersection had been inspected the day before the accident and found to be functioning properly.', \\\"The bus driver was operating the bus without a driver's license in violation of the State Motor Vehicle Code.\\\"]\",\"question\":\"A carpenter was driving home one night after a long day of work. He was driving his car north toward an intersection. There were stoplights at the intersection. The light for the carpenter was green, so the carpenter proceeded to drive through the intersection. A bus entered the intersection from the west and slammed into the carpenter's car. The carpenter was seriously injured in the crash and missed several weeks of work while recovering from the injuries. Which of the following facts or inferences, if true, would be most helpful in an action by the carpenter against the driver of the bus?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}