{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c3a1e1ee-d166-59da-b58b-d2fc6c59d33e","task_key":"all--test--c3a1e1ee-d166-59da-b58b-d2fc6c59d33e","task_revision_id":"1","upstream_id":"","short_description":"While parked on a downtown street, a police officer noticed a man sauntering…","config":"all","split":"test","body":"{\"choices\":\"[\\\"admitted, because the man's confession was not coerced.\\\", 'admitted, because the statement was not the product of interrogation.', \\\"excluded, because the officer's conduct violated the man's Sixth Amendment right to counsel.\\\", \\\"excluded, because the officer's conduct violated the man's attorney-client privilege.\\\"]\",\"question\":\"While parked on a downtown street, a police officer noticed a man sauntering down the street, whistling. As the man came under a street light, the officer recognized him as a parolee from a neighboring state. The officer got out, stood in front of the man, and asked him to give an account of himself. The man replied, \\\"I'm the president, you dope . . . get out of my way or the Secret Service will gun you down,\\\" and reached into his coat pocket to bring something out, which he held in his clenched fist. The officer forced the man's hand open and found a number of diamond rings therein. Just then, the police radio in the officer's car announced that the burglar alarm of a jewelry store had rung in police headquarters. Believing that the man was responsible for burglarizing the jewelry store, the officer arrested him and took him to the police station where he was booked and fingerprinted. After he was given his Miranda warnings, the man requested to speak to his attorney. The officer led the man to a telephone and asked him if he knew his attorney's phone number. The man responded and gave the officer a phone number. The officer dialed the number and waited until a voice answered, \\\"law offices. \\\" The officer then handed the telephone receiver to the man. As the officer was walking out of the room, he heard the man say, \\\"Hello, it's me. I just got arrested after robbing a jewelry store. \\\"At trial, the officer is called to testify to what the man told his attorney during their telephone conversation. Upon proper objection by the man's attorney, the officer's proposed testimony should be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}