{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5c56a986-a06a-5bb7-8a8b-2f6859c2deda","task_key":"default--test--1078","task_revision_id":"1","upstream_id":"1078","short_description":"A defendant was suspected of having burglarized his neighbor's apartment. The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admit the confession because it was voluntary, and the clothing because it was obtained pursuant to a valid search warrant.', 'suppress the confession because it was obtained in violation of his Fifth Amendment right to counsel, and the clothing because the search warrant was secured as a result of the confession.', \\\"suppress both the confession and the clothing because the defendant's right to counsel was violated.\\\", 'admit the confession because the defendant voluntarily accompanied the detectives, but suppress the clothing because the search warrant was based on the confession.', 'suppress the confession because the defendant failed to sign the waiver, but admit the clothing because it was obtained pursuant to a valid search warrant.', 'admit the confession because the defendant was not under arrest when he made it, and the clothing because it was obtained pursuant to a valid search warrant.', 'admit the confession because the question about the boots was about an unrelated robbery, but suppress the clothing because the search warrant was based on the confession.', \\\"suppress the confession because the defendant's attorney was not present, and the clothing because it was obtained pursuant to a valid search warrant.\\\", 'suppress the confession because the detectives continued questioning after the defendant invoked his right to counsel, but admit the clothing because it was obtained pursuant to a valid search warrant.', 'suppress the confession because it was obtained in violation of his Sixth Amendment right to counsel, but admit the clothing because it was obtained pursuant to a valid search warrant.']\",\"question\":\"A defendant was suspected of having burglarized his neighbor's apartment. The neighbor reported that his apartment had been ransacked and several items of clothing had been stolen. During the course of their investigation, two police detectives went to the defendant's place of work to interview him. After being advised of his Miranda rights, the defendant requested permission to call his attorney. Although his attorney was unavailable, the attorney's receptionist admonished him not to say anything. The defendant told the detectives he would have nothing further to say unless his attorney was present. The detectives then asked him if he would accompany them to the police station to answer some questions about an unrelated robbery. The defendant agreed. As they were driving to the police station, the defendant was asked where he had purchased the boots that he was wearing. He refused to answer. When they arrived at the police station, the detectives requested that he sign a waiver of his right to counsel. The defendant replied that he would not sign anything in the absence of his attorney. He was then placed in an interrogation room. Shortly thereafter, the defendant was told he could leave if he would be willing to say where he had purchased his boots. The defendant admitted that he had bought the boots and some shirts from a friend. From this information, the detectives obtained a search warrant and went to the defendant's home where they found the stolen clothing. The defendant is charged with burglarizing the apartment. At trial, the court should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}