{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5c064b20-f51f-5837-a9f9-b201e8d88056","task_key":"all--test--5c064b20-f51f-5837-a9f9-b201e8d88056","task_revision_id":"1","upstream_id":"","short_description":"A defendant was waiting in line to enter a movie theater when he noticed a free…","config":"all","split":"test","body":"{\"choices\":\"['larceny.', 'false pretenses.', 'forgery.', 'no crime.']\",\"question\":\"A defendant was waiting in line to enter a movie theater when he noticed a free movie pass on the ground. The pass had a space where the owner of the pass needed to write in his name. The defendant waited until he got to the cashier's window and presented the pass, seeking a free admission to the movie, and claiming that he received it as a birthday present from a friend. The cashier told him that the passes were invalid unless the holder's name was entered on the pass. The defendant wrote his name on the pass, gave it to the cashier, and was admitted to the theater. The defendant is guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}