{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0be40747-ccd6-5678-8f53-f289efb45207","task_key":"all--test--0be40747-ccd6-5678-8f53-f289efb45207","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The government dismantled the Chinese-style civil service system.', 'The government undertook a program of conquest in the Pacific.', 'The country sent emissaries to study western democracies and industry.', 'The country turned inward and closed its ports to all foreigners.']\",\"question\":\"This question refers to the following information.\\nThe condition of foreign states is not what it once was; they have invented the steamship, and introduced radical changes in the art of navigation. They have also built up their armies to a state of great efficiency and are possessed of war implements of great power and precision, in short have license to be formidable powers. If, therefore, we persistently cling to our antiquated systems, heaven only knows what a mighty calamity may befall our Empire.\\n—Adapted from a letter written by Lord Ii to Lord Tokugawa, 1847\\nHow did Japan respond to the concerns raised by Lord Ii during this era?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}