{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c22d5b29-f6e3-52a3-973a-809c66fb6a38","task_key":"all--test--c22d5b29-f6e3-52a3-973a-809c66fb6a38","task_revision_id":"1","upstream_id":"","short_description":"During a two-month span, there were 15 bank robberies in the city area. The…","config":"all","split":"test","body":"{\"choices\":\"['granted with respect to the ski mask, but denied with respect to the money.', 'granted with respect to the money, but denied with respect to the ski mask.', 'granted with respect to both the ski mask and the money.', 'denied with respect to both the ski mask and the money.']\",\"question\":\"During a two-month span, there were 15 bank robberies in the city area. The robberies were carried out by a single individual who entered the banks wearing a ski mask and carrying a machine gun. The police received information from a reliable informant that the defendant was the person responsible for the robberies. Acting on this tip, the police secured a valid arrest warrant and went to the defendant's home to apprehend him. When the officers rang the doorbell, the defendant opened the door and was placed under arrest. The defendant, who was wearing only underwear, asked the officers if he could put on a shirt and pants. One of the police officers asked the defendant if anyone else was present in the house. The defendant indicated that he was alone. The officer then accompanied the defendant to his bedroom to change his clothes. When the defendant asked permission to enter the closet for his pants, the officer told him that he needed to inspect the closet first. While the officer was inside the closet, he saw a ski mask that he recognized as part of the apparel worn by the bank robber. The officer confiscated the ski mask and then allowed the defendant inside the closet to change his clothes. While the officer and the defendant were in the bedroom, the officer's partner decided to conduct a protective sweep of the other rooms of the dwelling to see if any accomplices were present. Upon entering a back storage room, the other officer saw a box containing a large amount of $100 bills. The officer confiscated the money, which was later determined to be part of the money stolen during one of the bank robberies. The defendant is subsequently prosecuted for bank robbery. The prosecution seeks to introduce the ski mask and the money found in his home into evidence. A motion by the defendant's attorney to exclude the items will be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}