{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b5c24023-2618-5598-93be-c2e538cb8025","task_key":"all--test--b5c24023-2618-5598-93be-c2e538cb8025","task_revision_id":"1","upstream_id":"","short_description":"A man and a woman were arrested and charged with bank robbery. After receiving…","config":"all","split":"test","body":"{\"choices\":\"['denied, because the defendants did not have a reasonable expectation of privacy.', \\\"denied, because the statement did not violate the man's Fifth Amendment privilege against self-incrimination.\\\", 'granted, because both defendants requested an attorney before engaging in their conversation.', 'granted, because the police did not secure a warrant before installing the listening device.']\",\"question\":\"A man and a woman were arrested and charged with bank robbery. After receiving Miranda warnings, both requested an attorney. Following their arrests, the man and the woman were placed in separate jail cells. The next day before arraignment, the man asked a police officer if he could see his friend, the woman. The man was taken to the woman's jail cell and allowed to talk to her. Neither defendant knew that the police had placed a listening device in the woman's jail cell. As a result, their conversation was being overheard and recorded by the police. The police planted the listening device without first securing a warrant. While they were together, the man told the woman not to admit to anything because they could beat the charges with the help of a good lawyer. At trial, the prosecution sought to introduce the man's incriminating statement into evidence. The man's attorney has filed a motion to exclude. The motion should be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}