{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"42f1699d-2235-59ee-a9c3-4111c66863f9","task_key":"all--test--42f1699d-2235-59ee-a9c3-4111c66863f9","task_revision_id":"1","upstream_id":"","short_description":"A cashier worked part-time at a convenience store. As the cashier was sitting…","config":"all","split":"test","body":"{\"choices\":\"['Larceny.', 'Larceny by trick.', 'False pretenses.', 'No crime.']\",\"question\":\"A cashier worked part-time at a convenience store. As the cashier was sitting behind the cash register, he noticed a boy enter the store. While the boy walked through the store, the cashier watched him on a hidden camera located behind the cashier's counter. Thereupon, the cashier saw the boy pick up two packs of baseball cards and place them in his coat pocket. The boy then walked to the rear of the store, where he picked up a package of cupcakes. The boy took the cupcakes to the front counter and paid the cashier for the cupcakes only. The boy started to walk out of the store when the cashier said, \\\"Hey, kid, I'm not charging you for the baseball cards because you paid for the cupcakes. But next time I'm not going to let you get away with it. \\\" The baseball cards sold for $5 per pack. The boy has committed which, if any, crimes?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}