{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"469dcfed-c03f-53e4-be55-65509fcfcfe4","task_key":"all--test--469dcfed-c03f-53e4-be55-65509fcfcfe4","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"[\\\"The Tang Dynasty's strong bureaucracy at the local level\\\", \\\"The Tang Dynasty's large, standing professional army with no need of conscription\\\", 'Relatively advanced Tang agricultural technologies', 'The relatively high level of urbanization in Tang China']\",\"question\":\"This question refers to the following information.\\nI walk alongside the column, ask what's going on.\\nA soldier says simply: \\\"They call up more every day.\\n\\\"Some of us were sent north to the Yellow River at age fifteen,\\nAnd now at forty we're heading off to the garrisons in the west.\\nOn our first tour, the village headman had to tie our bandannas for us.\\nWhen we came back, our hair was white, but still there's more unrest.\\nThe frontier garrisons run with blood, enough to fill an ocean,\\nBut the Martial Emperor's territorial ambitions have yet to crest.\\nIn the hundred districts east of the mountains, throughout the land of Han,\\nThere must be ten thousand villages that brambles now infest.\\nEven if wives are strong enough to handle a hoe and plow,\\nThe crops grow every which way, the fields are all a mess.\\nIt's hardest for the Shanxi men, with their reputations as fighters:\\nThey're rounded up like dogs or chickens, every male impressed.\\n\\\"But sir, though it's good of you to ask,\\nComplaining isn't part of the soldier's task.\\nWe can only shake our heads. Take this winter:\\nThe Shanxi troops were never sent home.\\nThe District Officers are demanding the land tax,\\nBut where will it come from? You can't get blood from a stone!\\nI honestly think it's bad luck to bear a son now,\\nIt's better to have a daughter: at least she can marry\\nAnd live with the neighbors next door.\\nBut a son will end up lying on some distant prairie.\\\"\\nDu Fu, \\\"Ballad of the Army Carts,\\\" ca. 750 C.E.\\nWhich of the following does the poem provide direct evidence for in its discussion of taxation in the third stanza?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}