{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ab82ee49-f9e2-533f-af35-bd08b65b0375","task_key":"default--test--1644","task_revision_id":"1","upstream_id":"1644","short_description":"A man committed a series of heinous crimes, including a kidnapping, that took…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because the president's power to pardon can only be limited by a constitutional amendment.\\\", 'Yes, because the Supreme Court can overrule a presidential pardon.', 'No, because the Constitution does not allow for any limitations on the presidential pardon power.', 'Yes, under Article II', 'Yes, under Article I.', \\\"Yes, because the president's power to pardon is not unlimited according to the Supreme Court.\\\", 'Yes, under the Fifth Amendment.', 'Yes, because Congress can put limits on the pardon power.', \\\"No. While what the president did wasn't a good idea, the Constitution expressly grants the President an unqualified power to pardon.\\\", 'No, because the Constitution gives the president the absolute power to grant pardons for federal offenses.']\",\"question\":\"A man committed a series of heinous crimes, including a kidnapping, that took him across state lines and violated federal law. He was convicted and received a life sentence. The felon, however, was pardoned by the President of the United States, who was a childhood friend. A great hue and cry ensued, and the talking heads on television proclaimed that legislation was needed to prevent such injustice again. Would such legislation be constitutional?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}