{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c107e14-03a0-59ea-b035-0ca65ef2e5c3","task_key":"all--test--7c107e14-03a0-59ea-b035-0ca65ef2e5c3","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['an argument in favor of restrictions on immigration into the United States.', 'an attempt to unite working-class African American and white men and women.', 'an expression of black nationalism.', 'a pamphlet designed to promote the advancement of African Americans in industry.']\",\"question\":\"This question refers to the following information.\\n\\\"We are men; we have souls, we have passions, we have feelings, we have hopes, we have desires, like any other race in the world. The cry is raised all over the world today of Canada for the Canadians, of America for the Americans, of England for the English, of France for the French, of Germany for the Germans—do you think it is unreasonable that we, the Blacks of the world, should raise the cry of Africa for the Africans?\\\"\\n—Marcus Garvey, 1920\\nThe passage could best be understood as\",\"subject\":\"high_school_us_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}