{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"18c282e8-4efc-548a-b694-85d407501956","task_key":"all--test--18c282e8-4efc-548a-b694-85d407501956","task_revision_id":"1","upstream_id":"","short_description":"A boyfriend and his girlfriend were attending a house party. While the party was…","config":"all","split":"test","body":"{\"choices\":\"['factual impossibility.', 'mistake of fact.', 'entrapment.', 'no requisite intent.']\",\"question\":\"A boyfriend and his girlfriend were attending a house party. While the party was in progress, a few teenagers from the neighborhood vandalized some of the cars parked outside the house. They broke the headlights and stole the battery from the boyfriend's car. When the party ended, the boyfriend and his girlfriend left the house and got into his car. The boyfriend, who was about to drive his girlfriend home, was unaware of what had happened. He tried to start the car, but it wouldn't turn on. Two police officers, who were parked outside the house, watched the boyfriend as he tried to start the car. They then approached the boyfriend and charged him with attempting to violate a local ordinance making it a misdemeanor to drive at night without headlights. The boyfriend's best defense to the charge is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}