{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e03144f-25c5-5d9c-b086-468b851956d0","task_key":"all--test--2e03144f-25c5-5d9c-b086-468b851956d0","task_revision_id":"1","upstream_id":"","short_description":"A defendant worked as a patent attorney in a law firm. Late one evening, she…","config":"all","split":"test","body":"{\"choices\":\"['larceny only.', 'embezzlement only.', 'both larceny and embezzlement.', 'neither larceny nor embezzlement.']\",\"question\":\"A defendant worked as a patent attorney in a law firm. Late one evening, she left the office and walked outside to her car. While in the parking lot, she noticed a gold watch lying on the ground. When the defendant picked it up, she immediately identified the watch as belonging to one of the other lawyers in her law firm. She took it home with her, intending to return it the next day. The following morning, however, the defendant decided to keep the watch, and did so. The defendant has committed\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}