{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e3883c36-1a3f-57ae-b368-dca049f78a73","task_key":"all--test--e3883c36-1a3f-57ae-b368-dca049f78a73","task_revision_id":"1","upstream_id":"","short_description":"A federal grand jury was convened to investigate the alleged bribery by a…","config":"all","split":"test","body":"{\"choices\":\"['Under the separation of powers doctrine, the federal judiciary is without authority to review an assertion of executive privilege by the President.', 'The need for the confidentiality of high-level communications will sustain an absolute unqualified presidential privilege of immunity from judicial process on all occasions.', 'Article III does not vest the federal courts with power to resolve an intrabranch dispute.', 'Article II does not vest the President with an absolute, unqualified privilege to withhold evidence from a criminal prosecution.']\",\"question\":\"A federal grand jury was convened to investigate the alleged bribery by a foreign government of a special adviser to the President. The grand jury was probing the foreign government's efforts to obtain delivery of transport planes, which were embargoed by the State Department. The Justice Department was trying to ascertain whether the foreign government had offered bribes to the special adviser and other members of the U. S. government in order to secure delivery of the transport planes. The special adviser testified before the grand jury that both he and the President had several conferences with the foreign government's ambassador to the United States. He stated that during these meetings, they discussed problems in their region in general. He denied, however, any involvement in the foreign government's efforts to secure delivery of the transport planes. Two weeks after the special adviser testified, the grand jury returned an indictment, charging him and two other members of the President's staff with conspiracy to commit bribery and conspiracy to defraud the U. S. government. A special prosecutor was then appointed by the Justice Department to prepare the government's case. Upon motion by the special prosecutor, a subpoena duces tecum was issued directing the President to produce the minutes of his meetings with the special adviser and ambassador of the foreign government. The special prosecutor was able to determine the exact dates of the meetings through appointment records that had been previously subpoenaed. The President released several edited transcripts of these conversations. On the same day, the President's counsel filed a motion to quash the subpoena duces tecum, claiming an absolute executive privilege. Which of the following is the most accurate statement with regard to the President's claim of executive privilege?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}