{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c291230d-4fdb-50ec-b325-58abad6ccf43","task_key":"default--test--4980","task_revision_id":"1","upstream_id":"4980","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the American Revolution.', 'a negotiated settlement between the tribes of the Southeast and the U.S. government, by which the tribes were allowed to remain on their lands for ten years.', 'the forced removal of the Cherokee from their homeland.', 'the first Treaty of Fort Laramie, which guaranteed Indian possession of lands west of the Mississippi River.', 'peace between Indian tribes and white settlers.', 'the introduction of the Stamp Act.', 'the Treaty of Paris, ending the Revolutionary War.', 'the start of the French and Indian War.', 'the creation of the United States Constitution.', 'the formation of the United States Congress.']\",\"question\":\"This question refers to the following information.\\n\\\"I know that whenever the subject has occurred in conversation where I have been present, it has appeared to be the opinion of every one that we could not be taxed by a Parliament wherein we were not represented. But the payment of duties laid by an act of Parliament as regulations of commerce was never disputed. . . . An external tax is a duty laid on commodities imported; that duty is added to the first cost and other charges on the commodity, and, when it is offered for sale, makes a part of the price. If the people do not like it at that price, they refuse it; they are not obliged to pay it. But an internal tax is forced from the people without their consent if not laid by their own representatives. The Stamp Act says we shall have no commerce, make no exchange of property with each other, neither purchase nor grant, nor recover debts; we shall neither marry nor make our wills, unless we pay such and such sums; and thus it is intended to extort our money from us or ruin us by the consequence of refusing to pay it.\\\"\\n—Benjamin Franklin, Examination before Parliament, 1766\\nThe policy described most immediately led to\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}