{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a4d7b3e-0137-57db-ab25-c0a429eed397","task_key":"all--test--5a4d7b3e-0137-57db-ab25-c0a429eed397","task_revision_id":"1","upstream_id":"","short_description":"A defendant entered a bar and ordered a beer. The bartender politely told the…","config":"all","split":"test","body":"{\"choices\":\"['battery.', 'involuntary manslaughter.', 'assault with a deadly weapon.', 'murder.']\",\"question\":\"A defendant entered a bar and ordered a beer. The bartender politely told the defendant he would have to wait until two other patrons were served. The defendant became irate and pulled out a small penknife. Although the defendant intended only to frighten the bartender, he accidentally nicked the bartender's arm with the penknife. The bartender's arm became infected, and he died from gangrene two months later. The most serious crime that the defendant can be convicted of is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}