{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4678de5-912f-54de-8eab-9dd446fa5d26","task_key":"all--test--c4678de5-912f-54de-8eab-9dd446fa5d26","task_revision_id":"1","upstream_id":"","short_description":"An off-duty police officer was standing on a street corner waiting for a bus. A…","config":"all","split":"test","body":"{\"choices\":\"['assault with a deadly weapon.', 'involuntary manslaughter.', 'voluntary manslaughter.', 'murder.']\",\"question\":\"An off-duty police officer was standing on a street corner waiting for a bus. A man came up from behind and stole the police officer's wallet from his pants pocket. As the man was running away with the wallet, the police officer pulled out his service revolver. The police officer yelled at the man to stop and then fired several shots in the man's direction. The police officer did not aim directly at the man but shot at the pavement intending to frighten him. One of the bullets ricocheted off the sidewalk and struck the man, killing him. The police officer is guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}