{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f53f7ab8-a2c9-5ac8-819e-cc2990ffab16","task_key":"all--test--f53f7ab8-a2c9-5ac8-819e-cc2990ffab16","task_revision_id":"1","upstream_id":"","short_description":"A reliable police informant telephoned the police to report seeing two men in a…","config":"all","split":"test","body":"{\"choices\":\"['granted, as to the cocaine only.', 'granted, as to the marijuana only.', 'granted, as to both the cocaine and the marijuana.', 'denied, as to the cocaine and the marijuana.']\",\"question\":\"A reliable police informant telephoned the police to report seeing two men in a blue car selling narcotics outside a local movie theater. The informant gave a detailed description of the two men and the license number of the vehicle. A patrol car was immediately dispatched to the movie theater. A few minutes later, the police saw a car matching the description given parked about two blocks from the movie theater. When the police approached, they saw one man fitting the description provided. The police proceeded to the car and opened the doors of the vehicle. The police then pried open a locked tool case that was situated in the rear of the vehicle. Inside, the police found an envelope that contained about one gram of cocaine. They then placed the driver of the vehicle under arrest. Immediately thereafter, the police opened the glove compartment and found a small amount of marijuana. The driver of the van was subsequently prosecuted and charged with two counts of possession of controlled dangerous substances. He filed a motion to exclude the cocaine and marijuana from evidence. His motion will most likely be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}