{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2998a8c0-6fc1-52d2-9d30-0e6dc367a022","task_key":"all--test--2998a8c0-6fc1-52d2-9d30-0e6dc367a022","task_revision_id":"1","upstream_id":"","short_description":"Two executives from a company in China agreed to steal trade secrets of a…","config":"all","split":"test","body":"{\"choices\":\"['Yes, legal impossibility is a defense to criminal conspiracy.', 'Yes, factual impossibility is a defense to criminal conspiracy.', 'No, legal impossibility is not a defense to the crime of conspiracy.', 'No, the defendants were guilty of conspiracy as a matter of law.']\",\"question\":\"Two executives from a company in China agreed to steal trade secrets of a pharmaceutical company in the United States. They arranged a meeting to purchase the secrets from an unscrupulous employee of the pharmaceutical company in the company's New York office. The person selling the \\\"secrets\\\" was an undercover FBI agent. The envelope that he held, with the word \\\"Confidential\\\" typed on the outside of it, contained various recipes and gardening tips. When the defendants turned over the money, they were arrested on federal charges of conspiracy to steal trade secrets. The defendants moved to dismiss the indictment, claiming that it was legally impossible to steal trade secrets when in fact there were no trade secrets in the envelope. Will the federal district court likely grant the motion to dismiss?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}