{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fe4794f2-463a-5438-ad3e-3ba765ac733d","task_key":"default--test--1252","task_revision_id":"1","upstream_id":"1252","short_description":"During a two-month span, there were 15 bank robberies in the city area. The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['granted with respect to the money, but denied with respect to the ski mask.', 'granted, if the defendant can prove that the police did not have a valid warrant.', 'denied, if the prosecution can prove that the police had a valid warrant.', 'granted with respect to the money, if it can be proven that it was not obtained from the robberies.', 'granted with respect to the ski mask, but denied with respect to the money.', 'denied with respect to both the ski mask and the money.', 'denied with respect to the money, if it can be proven that it was obtained from the robberies.', 'granted with respect to the ski mask, if it can be proven that it was obtained illegally.', 'denied with respect to the ski mask, if it can be proven that it was used in the robberies.', 'granted with respect to both the ski mask and the money.']\",\"question\":\"During a two-month span, there were 15 bank robberies in the city area. The robberies were carried out by a single individual who entered the banks wearing a ski mask and carrying a machine gun. The police received information from a reliable informant that the defendant was the person responsible for the robberies. Acting on this tip, the police secured a valid arrest warrant and went to the defendant's home to apprehend him. When the officers rang the doorbell, the defendant opened the door and was placed under arrest. The defendant, who was wearing only underwear, asked the officers if he could put on a shirt and pants. One of the police officers asked the defendant if anyone else was present in the house. The defendant indicated that he was alone. The officer then accompanied the defendant to his bedroom to change his clothes. When the defendant asked permission to enter the closet for his pants, the officer told him that he needed to inspect the closet first. While the officer was inside the closet, he saw a ski mask that he recognized as part of the apparel worn by the bank robber. The officer confiscated the ski mask and then allowed the defendant inside the closet to change his clothes. While the officer and the defendant were in the bedroom, the officer's partner decided to conduct a protective sweep of the other rooms of the dwelling to see if any accomplices were present. Upon entering a back storage room, the other officer saw a box containing a large amount of $100 bills. The officer confiscated the money, which was later determined to be part of the money stolen during one of the bank robberies. The defendant is subsequently prosecuted for bank robbery. The prosecution seeks to introduce the ski mask and the money found in his home into evidence. A motion by the defendant's attorney to exclude the items will be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}