{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a901760e-9642-50fd-a259-eb3797861708","task_key":"all--test--a901760e-9642-50fd-a259-eb3797861708","task_revision_id":"1","upstream_id":"","short_description":"In which of the following situations is the defendant most likely to be found…","config":"all","split":"test","body":"{\"choices\":\"['Not intending to kill, a defendant hits a person over the head with a pipe. As a result, the person dies from head injuries.', 'Not intending to kill, a defendant throws a large chunk of ice off an overpass onto a busy freeway below. The chunk of ice crashes through a windshield, killing a driver.', 'Not intending to kill, a defendant kidnaps a victim and takes her to a remote area of the desert. The defendant then drives off, leaving victim alone in the sweltering heat after the ransom is not paid. The next day, the victim dies from exposure because she is unable to reach the nearest town eight miles away.', 'Not intending to kill, a defendant punches a victim in the face, causing the victim to fall backwards and strike his head on a curb, killing him.']\",\"question\":\"In which of the following situations is the defendant most likely to be found guilty of manslaughter rather than murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}