{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cf2ce4b3-fc33-549c-9668-4f54856f68b6","task_key":"default--test--1651","task_revision_id":"1","upstream_id":"1651","short_description":"The Latino American Party was founded and chartered in a state to promote the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"deny relief, because the Party's primary would be private action and not subject to restriction under the Constitution.\\\", 'grant relief, because the primary would be proscribed by the Thirteenth Amendment.', 'deny relief, because a pre-primary election is not within the scope of federal election control.', 'grant relief, because the primary would be proscribed by the Seventeenth Amendment.', 'grant relief, because the primary would be proscribed by the Fifteenth Amendment.', 'grant relief, because the primary would be proscribed by the First Amendment.', 'deny relief, because the primary would be upheld by the Nineteenth Amendment.', 'deny relief, because the political party has the right to choose its own members.', 'deny relief, because the primary would be upheld by the Tenth Amendment.']\",\"question\":\"The Latino American Party was founded and chartered in a state to promote the political power of Latino Americans. Members pledged themselves to vote only for candidates nominated by the party. At first, membership was open to any voter who pledged himself to those tenets, but after the defeats in a recent election, the Party expelled all nonLatino members upon the grounds that experience showed that only Latinos could be trusted to honor the obligation to vote only for the Party's nominees. Membership is currently confined to Latinos. In every election since, the Party's nominees have easily won election in their Congressional District. Subsequently, the Executive Committee proposes that the Party choose its own Latino candidate for governor of New Mexico in the upcoming election. It will put its slate for U. S. representatives, governor, and U. S. senator before Party members for mail ballot, along with any rival candidates who qualify for the poil. In an action by non-Latino citizens in the relevantCongressional District to enjoin the Latino AmericanParty from conducting the mail ballot primary, thefederal court will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}