{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cf273cc7-616c-59d3-aff4-eab1d48f86a3","task_key":"all--test--cf273cc7-616c-59d3-aff4-eab1d48f86a3","task_revision_id":"1","upstream_id":"","short_description":"A man was arrested and charged with robbery. Upon being taken into custody, he…","config":"all","split":"test","body":"{\"choices\":\"[\\\"inadmissible, because the videotape is the best evidence of the man's confession.\\\", 'inadmissible, because it is hearsay not within any recognized exception.', 'admissible, because the police officer had firsthand knowledge of the confession.', 'admissible, because the man was given his Miranda rights before the confession was elicited.']\",\"question\":\"A man was arrested and charged with robbery. Upon being taken into custody, he was given his Miranda rights and then taken to the police station for booking. At the stationhouse, the man told a police officer that he was prepared to make a confession. The police officer then turned on a video recorder and videotaped the man's confession. At trial, the prosecution called the police officer to testify to the incriminating statements that the man made in his confession. Upon objection by the man's attorney, the police officer's proposed testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}