{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"83bdf02d-b409-5493-87d2-3fef0c83161e","task_key":"all--test--83bdf02d-b409-5493-87d2-3fef0c83161e","task_revision_id":"1","upstream_id":"","short_description":"One afternoon, a woman was having lunch with her boss when the boss excused…","config":"all","split":"test","body":"{\"choices\":\"['No crime.', 'Larceny.', 'Embezzlement.', 'False pretenses.']\",\"question\":\"One afternoon, a woman was having lunch with her boss when the boss excused herself to go to the bathroom. As the boss stood up to leave the table, her wallet fell out of her pocketbook onto the floor. The boss was unaware of what occurred and proceeded to the bathroom. The woman, however, saw the wallet fall. Intending to steal it, the woman picked up the wallet and placed it in her pocket. Before the boss returned to the table, the woman had a change of heart and decided to give the wallet back. Thereupon, the woman told her boss what had happened and handed her the wallet when she returned from the bathroom. The woman is guilty of which, if any, crime?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}