{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bb796300-99de-5bbb-b8dc-53c35740f4c0","task_key":"all--test--bb796300-99de-5bbb-b8dc-53c35740f4c0","task_revision_id":"1","upstream_id":"","short_description":"One night, a defendant entered a liquor store, pointed a gun at the cashier, and…","config":"all","split":"test","body":"{\"choices\":\"[\\\"not guilty, because he was not the cause of the cashier's death.\\\", 'not guilty, because the subsequent prosecution violates double jeopardy.', 'not guilty, because the cashier decided to have the surgery, aware of the risks.', \\\"guilty, because he was the cause of the cashier's death.\\\"]\",\"question\":\"One night, a defendant entered a liquor store, pointed a gun at the cashier, and demanded the money from the register. The cashier was nervous and moving slowly, and when she accidentally spilled all of the money out of the bag and onto the floor, the defendant shot her and left. The bullet lodged close to the cashier's spine, paralyzing her. The defendant was subsequently prosecuted and convicted of robbery and assault with a deadly weapon. Two months later, the cashier decided she wanted an operation to remove the bullet. The doctors warned the cashier that the procedure was very risky and could result in her death. The cashier decided to go ahead with the surgery and, unfortunately, died as a result. If the defendant is now prosecuted for felony murder, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}