{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"59eea122-2386-5da9-b96b-24147120724f","task_key":"all--test--59eea122-2386-5da9-b96b-24147120724f","task_revision_id":"1","upstream_id":"","short_description":"In which of the following situations would the defendant most likely be found…","config":"all","split":"test","body":"{\"choices\":\"['In a dark alley, a defendant approached the victim and said, \\\"Give me the gold ring on your finger or I\\\\'ll shoot you with this gun. \\\" The victim gave him the ring. However, the ring really belonged to someone else, as the victim was just borrowing it for the evening to impress his girlfriend. In addition, the gun that the defendant had in his possession was really a water pistol.', 'A defendant broke into a house and took a stereo system. After he had placed the system in his car and was about to leave, the homeowner came home and saw him. She raced to the car and started to hit the defendant through the open window in an attempt to get her stereo back. The defendant punched her in the nose and drove away with the system.', 'A defendant was walking behind a shopper in a mall when he suddenly reached for her gold chain, pulled it from her neck, and ran away into the crowd. The shopper suffered a slight cut on her neck where the chain broke.', 'A defendant picked the lock on an apartment door. The noise startled the tenant, who had been sleeping. The defendant overpowered the tenant, tied him up, and forced him to disclose where he kept his money. The tenant told the defendant to look in the kitchen cabinet, which he did. The defendant found $120 in cash, took the money, and left the apartment.']\",\"question\":\"In which of the following situations would the defendant most likely be found NOT GUILTY of robbery?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}