{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b8b9265a-028a-5d31-b6dd-e861fee82c6b","task_key":"all--test--b8b9265a-028a-5d31-b6dd-e861fee82c6b","task_revision_id":"1","upstream_id":"","short_description":"A passenger on a subway car placed her pocketbook on the seat next to her and…","config":"all","split":"test","body":"{\"choices\":\"['larceny.', 'attempted larceny.', 'assault.', 'robbery.']\",\"question\":\"A passenger on a subway car placed her pocketbook on the seat next to her and began to zip up her jacket. Defendant was standing in front of the passenger and holding on to the hand rail. Seeing the pocketbook unattended, he suddenly grabbed it and tried to run away. As he did so, the passenger became very frightened and fell backward. She struck her head against the window and was knocked unconscious. In the commotion, Defendant dropped the pocketbook and hurriedly exited the subway car empty-handed. Defendant should be found guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}