{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"221f0238-e351-571f-920c-0ed9c4a9e48a","task_key":"default--test--4694","task_revision_id":"1","upstream_id":"4694","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['colonies', 'diplomatic embassies', 'outposts', 'vassal states', 'militarized zones', 'dominions', 'trading companies', 'protectorates', 'spheres of influence', 'businesses']\",\"question\":\"This question refers to the following information.\\n\\\"His Majesty the Emperor of China agrees, that British subjects, with their families and establishments, shall be allowed to reside, for the purposes of carrying on their mercantile pursuits, without molestation or restraint, at the cities and towns of Canton, Amoy, Foochowfoo, Ningpo, and Shanghai; and Her Majesty the Queen of Great Britain, &c., will appoint Superintendents, or Consular officers, to reside at each of the above-named cities or towns, to be the medium of communication between the Chinese authorities and the said merchants, and to see that the just duties and other dues of the Chinese Government, as hereafter provided for, are duly discharged by Her Britannic Majesty's subjects.\\\"\\nTreaty of Nanjing, 1839\\nThe concessions offered in this treaty most directly created British\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}