{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"08cede8a-504b-532f-b014-bb29af05af29","task_key":"all--test--08cede8a-504b-532f-b014-bb29af05af29","task_revision_id":"1","upstream_id":"","short_description":"A man was at a pub drinking beer and playing darts with a group of friends.…","config":"all","split":"test","body":"{\"choices\":\"['attempted robbery.', 'robbery.', 'larceny.', 'assault and larceny.']\",\"question\":\"A man was at a pub drinking beer and playing darts with a group of friends. Quite intoxicated, the man left the bar around midnight. He was staggering down the street when the defendant approached him from behind. The defendant took out a stick, stuck it against the man's back and told the man to give him his wallet or he was going to shoot him. Frightened, the man fainted and fell to the pavement. The defendant reached into the man's pants pocket and stole his wallet. The defendant is guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}