{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9faaed22-0173-54b9-8765-19c68c456300","task_key":"default--test--5026","task_revision_id":"1","upstream_id":"5026","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The establishment of a network of spies and informants among the nomadic tribes', 'Adroit diplomacy and establishment of client relationships with bordering nomads', 'The payment of tribute to nomadic rulers', 'The forced relocation of frontier peoples to central China', 'The use of economic incentives to secure the loyalty of nomadic rulers', 'The negotiation of peace treaties with nomadic rulers', 'The dissemination of Chinese culture and traditions among frontier peoples', 'The permanent settlement of Han Chinese in frontier zones', 'The ability of the Tang emperor to project military power on the frontier in order to impose his will']\",\"question\":\"This question refers to the following information.\\nI walk alongside the column, ask what's going on.\\nA soldier says simply: \\\"They call up more every day.\\n\\\"Some of us were sent north to the Yellow River at age fifteen,\\nAnd now at forty we're heading off to the garrisons in the west.\\nOn our first tour, the village headman had to tie our bandannas for us.\\nWhen we came back, our hair was white, but still there's more unrest.\\nThe frontier garrisons run with blood, enough to fill an ocean,\\nBut the Martial Emperor's territorial ambitions have yet to crest.\\nIn the hundred districts east of the mountains, throughout the land of Han,\\nThere must be ten thousand villages that brambles now infest.\\nEven if wives are strong enough to handle a hoe and plow,\\nThe crops grow every which way, the fields are all a mess.\\nIt's hardest for the Shanxi men, with their reputations as fighters:\\nThey're rounded up like dogs or chickens, every male impressed.\\n\\\"But sir, though it's good of you to ask,\\nComplaining isn't part of the soldier's task.\\nWe can only shake our heads. Take this winter:\\nThe Shanxi troops were never sent home.\\nThe District Officers are demanding the land tax,\\nBut where will it come from? You can't get blood from a stone!\\nI honestly think it's bad luck to bear a son now,\\nIt's better to have a daughter: at least she can marry\\nAnd live with the neighbors next door.\\nBut a son will end up lying on some distant prairie.\\\"\\nDu Fu, \\\"Ballad of the Army Carts,\\\" ca. 750 C.E.\\nBased on the poem and your knowledge of history, Tang relations with western nomadic peoples and frontier peoples relied on which of the following?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}