{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"125b2bc2-b48c-569e-830a-b36a6e92b759","task_key":"default--test--4808","task_revision_id":"1","upstream_id":"4808","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Women in Tang society had limited roles outside of the home.', 'Imperial rescripts emphasized the birth of able sons over daughters to fulfill conscription quotas.', 'Agricultural work could be shared by women and men alike.', 'Women were encouraged to become soldiers to fill the conscription quotas.', 'Traditional Chinese gender relations were reversed during the reign of Empress Wu Zetian.', 'The birth of a daughter was seen as a misfortune in Tang society.', 'Men were predominantly responsible for the household while women worked in the fields.', 'Confucian relationships between husband and wife were still strong in Tang society.']\",\"question\":\"This question refers to the following information.\\nI walk alongside the column, ask what's going on.\\nA soldier says simply: \\\"They call up more every day.\\n\\\"Some of us were sent north to the Yellow River at age fifteen,\\nAnd now at forty we're heading off to the garrisons in the west.\\nOn our first tour, the village headman had to tie our bandannas for us.\\nWhen we came back, our hair was white, but still there's more unrest.\\nThe frontier garrisons run with blood, enough to fill an ocean,\\nBut the Martial Emperor's territorial ambitions have yet to crest.\\nIn the hundred districts east of the mountains, throughout the land of Han,\\nThere must be ten thousand villages that brambles now infest.\\nEven if wives are strong enough to handle a hoe and plow,\\nThe crops grow every which way, the fields are all a mess.\\nIt's hardest for the Shanxi men, with their reputations as fighters:\\nThey're rounded up like dogs or chickens, every male impressed.\\n\\\"But sir, though it's good of you to ask,\\nComplaining isn't part of the soldier's task.\\nWe can only shake our heads. Take this winter:\\nThe Shanxi troops were never sent home.\\nThe District Officers are demanding the land tax,\\nBut where will it come from? You can't get blood from a stone!\\nI honestly think it's bad luck to bear a son now,\\nIt's better to have a daughter: at least she can marry\\nAnd live with the neighbors next door.\\nBut a son will end up lying on some distant prairie.\\\"\\nDu Fu, \\\"Ballad of the Army Carts,\\\" ca. 750 C.E.\\nThe poem provides evidence of which of the following with respect to gender relations in Tang society?\",\"src\":\"ori_mmlu-high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}