{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bd73e325-2ee3-56ea-b940-a598f8892481","task_key":"all--test--bd73e325-2ee3-56ea-b940-a598f8892481","task_revision_id":"1","upstream_id":"","short_description":"Police received information from a reliable informant that a defendant would be…","config":"all","split":"test","body":"{\"choices\":\"['granted, because the police did not secure a warrant before installing the eavesdropping device.', 'granted, because the police did not obtain prior approval of the telephone company before installing the eavesdropping device.', 'denied, because the defendant did not have reasonable expectation of privacy.', 'denied, because the police had probable cause based on the information supplied by the informant.']\",\"question\":\"Police received information from a reliable informant that a defendant would be extorting money from a victim. The informant gave the police explicit details about the extortion scheme: the defendant would be calling the victim from a public phone booth on a particular street corner on a certain date and time to make the extortionist demand. Without notif'ing the telephone company or seeking a warrant, the police installed an eavesdropping device on the public phone. At the time on the date specified by the informant, the defendant phoned the victim from the telephone booth and said, \\\"You better pay me $5,000 by next week or I'll kill you and your family. \\\" The defendant provided additional details about how and where to make the payoff. Based on the taped conversation, the police arrested the defendant and charged him with extortion. Prior to trial, the defendant's attorney moves to exclude the taped conversation from evidence. The defense motion should be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}