{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"da376684-a1af-5494-9520-c7974e9091bf","task_key":"all--test--da376684-a1af-5494-9520-c7974e9091bf","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The debate between interventionism and isolationism in foreign policy.', 'The debate between separatism and integration when it came to the place of African Americans in American society.', 'The debate between exclusion and inclusion when it came to immigration policy.', 'The debate between laissez-faire policies and government intervention in economic affairs.']\",\"question\":\"This question refers to the following information.\\n\\\"We are men; we have souls, we have passions, we have feelings, we have hopes, we have desires, like any other race in the world. The cry is raised all over the world today of Canada for the Canadians, of America for the Americans, of England for the English, of France for the French, of Germany for the Germans—do you think it is unreasonable that we, the Blacks of the world, should raise the cry of Africa for the Africans?\\\"\\n—Marcus Garvey, 1920\\nThe passage above presents a position in which of the following ongoing debates in American history?\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}