{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c0a59bd1-511c-5229-886f-7356a9635b92","task_key":"default--test--4877","task_revision_id":"1","upstream_id":"4877","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['a colony that banned all forms of religious expression.', 'one of the least religiously diverse colonies in America.', 'one of the most religiously diverse colonies in British America.', 'known for its strict adherence to a single religious doctrine.', 'known for its hostility to traditional religious practices.', 'a colony that forbade the practice of any religion other than Christianity.', 'a colony known for religious repression and intolerance.', 'notorious for witch hunting and popular superstition.', 'the only colony in America where religion was not practiced.', 'a colony where religious diversity was punishable by law.']\",\"question\":\"This question refers to the following information.\\nBECAUSE no People can be truly happy, though under the greatest Enjoyment of Civil Liberties, if abridged of the Freedom of their Consciences, as to their Religious Profession and Worship: And Almighty God being the only Lord of Conscience, Father of Lights and Spirits; and the Author as well as Object of all divine Knowledge, Faith and Worship, who only doth enlighten the Minds, and persuade and convince the Understanding of People, I do hereby grant and declare, That no Person or Persons, inhabiting in this Province or Territories, who shall confess and acknowledge One almighty God, the Creator, Upholder and Ruler of the World; and profess him or themselves obliged to live quietly under the Civil Government, shall be in any Case molested or prejudiced, in his or their Person or Estate, because of his or their conscientious Persuasion or Practice, nor be compelled to frequent or maintain any religious Worship, Place or Ministry, contrary to his or their Mind.…\\n—William Penn, Charter of Privileges Granted by William Penn,\\nesq. to the Inhabitants of Pennsylvania and Territories, October 28, 1701\\nBecause of Penn's Charter of Privileges, Pennsylvania became\",\"src\":\"ori_mmlu-high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}