{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4c2e7e6-85bb-5221-966b-dc6996ca4958","task_key":"default--test--1194","task_revision_id":"1","upstream_id":"1194","short_description":"Police received information from a reliable informant that a defendant would be…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['granted, because the police should have obtained a court order before recording the conversation.', 'denied, because the police had probable cause based on the information supplied by the informant.', 'denied, because the defendant did not have reasonable expectation of privacy.', \\\"granted, because the police violated the defendant's rights by not informing him of the eavesdropping device.\\\", 'denied, because the information from the informant was reliable and specific.', 'granted, because the police did not obtain prior approval of the telephone company before installing the eavesdropping device.', 'denied, because the threat made by the defendant was a serious crime.', 'granted, because the police did not secure a warrant before installing the eavesdropping device.', 'granted, because the defendant was not aware of the eavesdropping device.', 'denied, because the phone booth is a public place with no expectation of privacy.']\",\"question\":\"Police received information from a reliable informant that a defendant would be extorting money from a victim. The informant gave the police explicit details about the extortion scheme: the defendant would be calling the victim from a public phone booth on a particular street corner on a certain date and time to make the extortionist demand. Without notif'ing the telephone company or seeking a warrant, the police installed an eavesdropping device on the public phone. At the time on the date specified by the informant, the defendant phoned the victim from the telephone booth and said, \\\"You better pay me $5,000 by next week or I'll kill you and your family. \\\" The defendant provided additional details about how and where to make the payoff. Based on the taped conversation, the police arrested the defendant and charged him with extortion. Prior to trial, the defendant's attorney moves to exclude the taped conversation from evidence. The defense motion should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}