{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ac8112da-7ffa-5786-a499-1f89fb8b4759","task_key":"all--test--ac8112da-7ffa-5786-a499-1f89fb8b4759","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Meritocratic appointments by rulers to their bureaucracies', 'Long-distance contact between Egypt and other lands', 'The clan as the basic political unit', 'A lack of emphasis on martial ability']\",\"question\":\"This question refers to the following information.\\nThis ruler of Syria made me spend many years as commander of his army,\\nEvery land to which I turned\\nI overcame.\\nI destroyed its green fields and its wells,\\nI captured its cattle, I took captive its inhabitants, I deprived them of their provisions,\\nand I slew [many] people…by my sword, my bow, my marchings, and my good devices.\\nThus my excellence was in his heart; he loved me and he knew my valor;\\n…he set me at the head of his sons, when he saw the success of my handiwork.\\nThere came a champion of Syria\\nto defy me in my tent;\\na bold man without equal, for he had vanquished all his rivals.\\nHe said, \\\"Let Sanehat fight with me.\\\"\\nHe thought to overcome me; he designed to take my cattle, thus being counseled by his tribe.\\nThe Tale of Sanehat, Egyptian poem written during the Middle Kingdom, ca.1800 B.C.E.\\nThe text of this passage is best seen as evidence of which of the following in Egyptian society?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}