{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"206ede19-bd6f-50fa-b744-ab365421eff4","task_key":"all--test--206ede19-bd6f-50fa-b744-ab365421eff4","task_revision_id":"1","upstream_id":"","short_description":"A man committed a series of heinous crimes, including a kidnapping, that took…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No. While what the president did wasn't a good idea, the Constitution expressly grants the President an unqualified power to pardon.\\\", 'Yes, under Article I.', 'Yes, under Article II', 'Yes, because Congress can put limits on the pardon power.']\",\"question\":\"A man committed a series of heinous crimes, including a kidnapping, that took him across state lines and violated federal law. He was convicted and received a life sentence. The felon, however, was pardoned by the President of the United States, who was a childhood friend. A great hue and cry ensued, and the talking heads on television proclaimed that legislation was needed to prevent such injustice again. Would such legislation be constitutional?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}