{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4dc5f2ad-b907-545d-a890-69aa104d18f8","task_key":"all--test--4dc5f2ad-b907-545d-a890-69aa104d18f8","task_revision_id":"1","upstream_id":"","short_description":"A defendant went to a party at his friend's house. While he was there, the…","config":"all","split":"test","body":"{\"choices\":\"['Larceny.', 'Larceny and burglary.', 'Larceny and attempted extortion.', 'Robbery and extortion.']\",\"question\":\"A defendant went to a party at his friend's house. While he was there, the defendant drank a lot of beers and countless shots. The defendant eventually stumbled into the bathroom where he spotted a purse lying on the floor under the sink. Wondering vaguely if it might contain more beer, he opened it up and found his friend's wallet. He grabbed some money from the wallet and left the bathroom. As he was descending the stairs with the money in his hand, his friend spotted him. She was initially concerned he might tumble down the stairs but upon seeing the money in his hand, she asked him where he got it. The defendant responded, \\\"Ha! Your money or your life!\\\" Not taking him seriously, the friend let him pass by. It wasn't until later that she realized that the defendant had taken the money from her wallet. The defendant should be charged with which of the following crimes?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}