{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2f9bc553-fed6-5023-9d46-c155614b7363","task_key":"default--test--1895","task_revision_id":"1","upstream_id":"1895","short_description":"Law enforcement authorities had probable cause to suspect a man of murder.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, the entry was a violation of the knock and announce rule.', 'No, the entry did not require an arrest warrant because the authorities already had established probable cause.', \\\"Yes, the police must obtain an arrest warrant to enter someone's home for an arrest.\\\", 'No, the police did not need a warrant because the gun was in plain view.', 'Yes, the police needed both a search warrant for the gun and an arrest warrant for the suspect.', 'No, because the suspect was not home at the time of entry, no warrant was needed.', \\\"Yes, the police violated the suspect's right to privacy by forcibly entering his home.\\\", 'Yes, the police needed a search warrant in addition to an arrest warrant to seize the gun.', 'No, the entry was authorized by exigent circumstances and the gun was in plain view.', 'No, the police were in the right because they announced their presence before entering.']\",\"question\":\"Law enforcement authorities had probable cause to suspect a man of murder. Without an arrest warrant, they went to his home to arrest him. They knocked and announced, and hearing no response, forcibly broke open the door and entered the house. They didn't find anyone but seized a handgun that was in plain view. The suspect was later arrested for murder; he filed for suppression of the gun. Will the suppression court throw out the gun?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}