{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b2ff6a38-4619-5cae-a67d-2d741c554c20","task_key":"all--test--b2ff6a38-4619-5cae-a67d-2d741c554c20","task_revision_id":"1","upstream_id":"","short_description":"A defendant met her friend at the electronics store where the friend worked.…","config":"all","split":"test","body":"{\"choices\":\"['receiving stolen property.', 'larceny.', 'receiving stolen property and larceny.', 'neither receiving stolen property nor larceny.']\",\"question\":\"A defendant met her friend at the electronics store where the friend worked. Unbeknownst to the defendant, her friend had taken a music player from the display case and stuffed it in her bag earlier in the day. Knowing that employees are subject to search when they leave the store, the friend surreptitiously put the music player into the defendant's bag. The defendant and her friend proceeded to a local tavern and drank shots of tequila. The friend became intoxicated and forgot to retrieve the stolen music player from the defendant's bag. After leaving the tavern, the defendant proceeded home. Upon opening her bag, she was surprised to see the music player. Believing that the music player had accidentally fallen into her bag at the bar, the defendant planned to return it the next day. The following day, however, the defendant changed her mind and decided to keep it. The defendant is guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}