{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1c2f7ec-f959-58ea-8724-cb20d722462e","task_key":"all--test--f1c2f7ec-f959-58ea-8724-cb20d722462e","task_revision_id":"1","upstream_id":"","short_description":"One night, a victim was in her home when she heard the doorbell ring. When the…","config":"all","split":"test","body":"{\"choices\":\"['Assault, battery, and robbery.', 'Larceny, robbery, and burglary.', 'Robbery and burglary.', 'Robbery only.']\",\"question\":\"One night, a victim was in her home when she heard the doorbell ring. When the victim opened the door, she was confronted by three defendants. They pushed the victim inside her house and threatened her with bodily harm if she didn't cooperate. After tying her up with ropes, the trio then proceeded to ransack the victim's home. They then placed many of the victim's valuables and other possessions into a large sack and hurriedly left her house. What crimes should the defendants be convicted of?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}