{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"58437214-18bf-5b77-a4dc-a8194d9c6a25","task_key":"default--test--4849","task_revision_id":"1","upstream_id":"4849","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['made it too difficult for the government to raise money through taxes and duties', 'did not include a provision for creating new states', 'created a too-powerful chief executive', 'established a single, unified national identity too quickly', 'denied the federal government the power to mediate disputes between states', 'granted too much power to the judicial branch', 'failed to establish a national currency', 'did not include a mechanism for their own amendment', 'did not provide for a national army or navy', 'allowed for the unrestricted movement of goods and people between states']\",\"question\":\"This question refers to the following information.\\n\\\"Permit us, then, earnestly to recommend these articles to the immediate and dispassionate attention of the legislatures of the respective states. Let them be candidly reviewed under a sense of the difficulty of combining in one system the various sentiments and interests of a continent divided into so many sovereign and independent communities, under a conviction of the absolute necessity of uniting all our councils and all our strength, to maintain and defend our common liberties….\\\"\\nJournals of the Continental Congress, 1777\\nA major weakness of the Articles of Confederation was that they\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}