{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1ee19b73-71ea-5453-8207-f5ecf3fa5d76","task_key":"all--test--1ee19b73-71ea-5453-8207-f5ecf3fa5d76","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['British support for growing resistance movements in Eastern Europe', 'British trade deals with American manufacturers of military hardware', 'British appeasement of the Axis powers', 'British fears about a possible invasion attempt by Nazi Germany']\",\"question\":\"This question refers to the following information.\\nSource 1:\\n\\\"It is impossible to demand that an impossible position should be cleared up by peaceful revision and at the same time constantly reject peaceful revision. It is also impossible to say that he who undertakes to carry out these revisions for himself transgresses a law, since the Versailles \\\"Diktat\\\" is not law to us. A signature was forced out of us with pistols at our head and with the threat of hunger for millions of people. And then this document, with our signature, obtained by force, was proclaimed as a solemn law.\\\"\\nAdolf Hitler, speech to the Reichstag, September 1, 1939\\nSource 2:\\n\\\"We shall not flag or fail. We shall go on to the end. We shall fight in France, we shall fight on the seas and the oceans, we shall fight with growing confidence and growing strength in the air, we shall defend our island, whatever the cost may be. We shall fight on the beaches, we shall fight on the landing grounds, we shall fight in the fields and in the streets, we shall fight in the hills; we shall never surrender.\\\"\\nWinston Churchill, speech before Parliament, June 4, 1940\\nWinston Churchill's speech in Source 2 is best understood in the context of which of the following?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}