{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"17f7ab6e-aef9-5de9-b1f9-0dac8f976336","task_key":"all--test--17f7ab6e-aef9-5de9-b1f9-0dac8f976336","task_revision_id":"1","upstream_id":"","short_description":"Government drug officials from a neighboring country contacted a federal agency…","config":"all","split":"test","body":"{\"choices\":\"['granted, because the police failed to obtain a search warrant before opening the trunk of the vehicle.', 'granted, because the police failed to obtain a search warrant before opening the package.', 'denied, because the police had probable cause to conduct the search.', 'denied, because the search was incident to a lawful arrest.']\",\"question\":\"Government drug officials from a neighboring country contacted a federal agency notifying the agency that drugs were being shipped by express mail next-day service to a defendant. The federal agency contacted the local police with the information regarding the defendant. The local police had long suspected the defendant of dealing drugs. The next day, two undercover police officers drove to the defendant's home and waited outside. A couple of hours later, the officers saw an express mail delivery truck pull up outside the defendant's home. The delivery person carried an express mail box to the front door and rang the doorbell. The officers then witnessed the defendant answer the door and take the package inside. Approximately 20 minutes later, the defendant left the house carrying the express mail box. She walked to her car, placed the package in the trunk, and drove off. The police followed in the squad car and pulled her over at the next traffic signal. The officers placed the defendant under arrest and instructed her to open the trunk. They confiscated the express mail box and opened it. Inside was a package of white powder, which lab tests later confirmed to be cocaine. The defendant was charged with illegal possession of cocaine. While awaiting trial, her attorney filed a motion to suppress the introduction of the cocaine into evidence. The motion will be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}