{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ed28db1f-eebe-5977-a2a0-0ab262ce569f","task_key":"all--test--ed28db1f-eebe-5977-a2a0-0ab262ce569f","task_revision_id":"1","upstream_id":"","short_description":"A small town police officer pulled over a driver for speeding. He believed that…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the search was unconstitutional due to the officer having no reasonable suspicion that would justify searching the car.', 'Yes, because when a car is pulled over for speeding, the officer must always obtain a search warrant prior to making any search.', 'No, the stop and the search were within the normal bounds of propriety for a speeding stop.', 'No, because driver being fidgety is enough for a probable cause full search of the vehicle.']\",\"question\":\"A small town police officer pulled over a driver for speeding. He believed that the driver was acting irritable and fidgety, but he had no articulable reason to think anything was wrong. He searched the car anyway and found two cartons of freshly canned peaches, which were owned by the driver's neighbor and reported stolen off of her porch 24 hours earlier. Authorities charged him with theft under the state criminal code. His motion to suppress the evidence because of an unlawful search was denied. On appeal, will the appellate court likely reverse the lower court decision denying the motion to suppress?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}