{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c82b1052-fdda-570e-916d-cb8ae598389c","task_key":"all--test--c82b1052-fdda-570e-916d-cb8ae598389c","task_revision_id":"1","upstream_id":"","short_description":"A man is at home in his apartment, alone, late at night. Someone repeatedly…","config":"all","split":"test","body":"{\"choices\":\"['Burglary and attempted murder.', 'Robbery and attempted murder.', 'Robbery, burglary and attempted murder.', 'Robbery and burglary.']\",\"question\":\"A man is at home in his apartment, alone, late at night. Someone repeatedly buzzes his apartment ringer outside the building, but he doesn't answer it because he is very tired and wants to sleep. Later he hears the door to his apartment being kicked in and he is immediately accosted by a man with a gun, who waves the gun at him and says, \\\"You are getting me your cash or else.\\\" The man opens his safe and turns over the money. The perpetrator absconds with the money but is picked up by the police a few minutes later. What common law crimes has the perpetrator probably committed?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}