{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07cc30cb-b807-5f13-b0ac-959af2117edd","task_key":"all--test--07cc30cb-b807-5f13-b0ac-959af2117edd","task_revision_id":"1","upstream_id":"","short_description":"A man entered a store with a gun, planning to rob the store. He was unfamiliar…","config":"all","split":"test","body":"{\"choices\":\"['No.', 'Yes, but only for the death of the store manager.', 'Yes, but only for the death of the clerk.', 'Yes, for the deaths of both the manager and the clerk.']\",\"question\":\"A man entered a store with a gun, planning to rob the store. He was unfamiliar with guns and thought that the safety lock was on. When the store manager refused to give him any money, the man shook the gun at her to scare her into cooperating. The gun fired, killing the manager. A clerk ran in from the back stockroom. The man fired a shot into the air to scare the clerk. Unbeknownst to the man, the clerk had a heart condition, and the fright caused her to suffer a fatal heart attack. Can the man properly be convicted of felony murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}