{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2af8a130-cd9e-5a2a-9b90-f402e5510896","task_key":"all--test--2af8a130-cd9e-5a2a-9b90-f402e5510896","task_revision_id":"1","upstream_id":"","short_description":"Max entered a mini-mart store, pulled out a gun, and demanded that the cashier…","config":"all","split":"test","body":"{\"choices\":\"['Acquitted, because he lacked the necessary intent required for first degree murder.', \\\"Acquitted, because the cashier's actions were the proximate cause of the death.\\\", 'Convicted, because any death that is a natural and probable consequence of a felony is first degree murder.', 'Convicted, because intent to commit serious bodily harm is assumed when a deadly weapon is used.']\",\"question\":\"Max entered a mini-mart store, pulled out a gun, and demanded that the cashier empty the register. As he nervously looked around for police, the cashier reached out and attempted to grab the gun. Max then droped the gun, which accidentally discharged, killing a nearby customer. Max is charged with first degree murder. He will likely be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}