{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fb762dbd-34cf-5ab0-b694-9a8afcf47d9b","task_key":"default--test--1556","task_revision_id":"1","upstream_id":"1556","short_description":"One evening, an undercover narcotics agent, with the help of a confidential…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the drug dealer has a right to face his accuser.', 'Yes, because informants would not assist in crime prevention unless they were reasonably certain that their identities would be kept secret.', 'Yes, because revealing the identity of the informant could jeopardize ongoing investigations.', 'No, because under the Fifth Amendment, a defendant has the right to be confronted by witnesses against him.', 'Yes, because an informant has a Fifth Amendment privilege against self-incrimination.', \\\"No, because the informant's testimony could be crucial for the defense's entrapment claim.\\\", 'Yes, because the man was a fugitive and the government could not have served a subpoena.', \\\"Yes, because the informant's safety could be at risk if his identity is revealed.\\\", 'No, because under the Sixth Amendment, a defendant has the right to a fair trial.', \\\"No, because the government does not have the right to protect an informant's identity.\\\"]\",\"question\":\"One evening, an undercover narcotics agent, with the help of a confidential informant, met with a man. The man took the agent to the home of a drug dealer, who sold the agent four pounds of marijuana. At trial, the drug dealer claimed the defense of entrapment and demanded that the informant's identity be disclosed and that he be produced. At no time prior to trial did the drug dealer seek to subpoena the man. However, the man was a fugitive from justice the whole time, and no subpoena could have been served. The government claims a right to protect the informant's identity. Should the government's claim be honored?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}