{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c61af8aa-d35f-5877-a90c-28322444f598","task_key":"default--test--4812","task_revision_id":"1","upstream_id":"4812","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Spanish desired to convert the Barbary states to Catholicism.', 'The Spanish believed Portugal did not know about their discovery of the New World.', 'The Spanish wanted to preserve the independence of Native states in the New World as a buffer against Portuguese expansion.', 'The Spanish thought India would be harder to conquer than the New World.']\",\"question\":\"This question refers to the following information.\\n\\\"Thereupon it was declared by the above-mentioned representatives of the aforesaid King and Queen of Castile, Leon, Aragon, Sicily, Granada, etc., and of the aforesaid King of Portugal and the Algarves, etc.:\\n[I.] That, whereas a certain controversy exists between the said lords, their constituents, as to what lands, of all those discovered in the ocean sea up to the present day, the date of this treaty, pertain to each one of the said parts respectively; therefore, for the sake of peace and concord, and for the preservation of the relationship and love of the said King of Portugal for the said King and Queen of Castile, Aragon, etc., it being the pleasure of their Highnesses, they, their said representatives, acting in their name and by virtue of their powers herein described, covenanted and agreed that a boundary or straight line be determined and drawn north and south, from pole to pole, on the said ocean sea, from the Arctic to the Antarctic pole. This boundary or line shall be drawn straight, as aforesaid, at a distance of three hundred and seventy leagues west of the Cape Verde Islands, being calculated by degrees, or by any other manner as may be considered the best and readiest, provided the distance shall be no greater than abovesaid. And all lands, both islands and mainlands, found and discovered already, or to be found and discovered hereafter, by the said King of Portugal and by his vessels on this side of the said line and bound determined as above, toward the east, in either north or south latitude, on the eastern side of the said bound provided the said bound is not crossed, shall belong to, and remain in the possession of, and pertain forever to, the said King of Portugal and his successors. And all other lands, both islands and mainlands, found or to be found hereafter, discovered or to be discovered hereafter, which have been discovered or shall be discovered by the said King and Queen of Castile, Aragon, etc., and by their vessels, on the western side of the said bound, determined as above, after having passed the said bound toward the west, in either its north or south latitude, shall belong to, and remain in the possession of, and pertain forever to, the said King and Queen of Castile, Leon, etc., and to their successors.\\\"\\nTreaty of Tordesillas, 1494\\nWhich of the following inferences best explains the reason the King and Queen of Castile and Aragon insisted on Spanish control of territory west of the demarcation line?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}