{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"61f4dbd4-5ef5-5f4f-bf37-acb3fde37517","task_key":"all--test--61f4dbd4-5ef5-5f4f-bf37-acb3fde37517","task_revision_id":"1","upstream_id":"","short_description":"One evening, an undercover narcotics agent, with the help of a confidential…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because an informant has a Fifth Amendment privilege against self-incrimination.', 'Yes, because informants would not assist in crime prevention unless they were reasonably certain that their identities would be kept secret.', 'No, because under the Fifth Amendment, a defendant has the right to be confronted by witnesses against him.', 'No, because under the Sixth Amendment, a defendant has the right to a fair trial.']\",\"question\":\"One evening, an undercover narcotics agent, with the help of a confidential informant, met with a man. The man took the agent to the home of a drug dealer, who sold the agent four pounds of marijuana. At trial, the drug dealer claimed the defense of entrapment and demanded that the informant's identity be disclosed and that he be produced. At no time prior to trial did the drug dealer seek to subpoena the man. However, the man was a fugitive from justice the whole time, and no subpoena could have been served. The government claims a right to protect the informant's identity. Should the government's claim be honored?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}