{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ba75b197-7dc4-5648-bcc2-3a0513ee1f39","task_key":"all--test--ba75b197-7dc4-5648-bcc2-3a0513ee1f39","task_revision_id":"1","upstream_id":"","short_description":"A regular customer at a lounge is aware that the owner customarily keeps a gun…","config":"all","split":"test","body":"{\"choices\":\"['attempted larceny.', 'attempted robbery.', 'assault.', 'assault and attempted larceny.']\",\"question\":\"A regular customer at a lounge is aware that the owner customarily keeps a gun hidden behind the bar. One night the customer was sitting at the bar and ordered a club soda. After serving him, the owner walked to the other end of the bar to wait on another customer. As he did so, the customer leaned over the bar and grabbed the gun. The customer pointed the gun at the owner and said, \\\"This is a robbery. . . hand over all your money. \\\"Knowing the gun was unloaded, the owner refused to comply. The customer dropped the gun and ran out of the bar. Besides larceny of the gun, the customer will be convicted of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}