{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4467534e-6609-553e-a5c7-d0df0144c410","task_key":"all--test--4467534e-6609-553e-a5c7-d0df0144c410","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['The election of F. W. de Klerk', 'The election of Nelson Mandela', 'The bombing of Lesotho', 'The Boer War']\",\"question\":\"This question refers to the following information.\\nThe following poem refers to an incident in Sharpeville, South Africa, in 1960. To protest the requirement that they carry papers documenting their identity and residence, black South Africans gathered in front of police stations without their papers. Though other protests ended peacefully, in Sharpeville, police fired into the crowd.\\nWhat is important\\nabout Sharpeville\\nis not that seventy died:\\nnor even that they were shot in the back\\nretreating, unarmed, defenceless\\nand certainty not\\nthe heavy caliber slug\\nthat tore through a mother's back\\nand ripped through the child in her arms\\nkilling it\\nRemember Sharpeville\\nbullet-in-the-back day\\nBecause it epitomized oppression\\nand the nature of society\\nMore clearly than anything else;\\nit was the classic event\\nNowhere is racial dominance\\nmore clearly defined\\nnowhere the will to oppress\\nmore clearly demonstrated\\nWhat the world whispers\\napartheid declares with snarling guns\\nthe blood the rich lust after\\nSouth Africa spills dust\\nRemember Sharpeville\\nRemember bullet-in-the-back day\\nAnd remember the unquenchable will for freedom\\nRemember the dead\\nand be glad\\n—Dennis Brutus, 1973\\nWhat event symbolized the end of South Africa's institutionalized discrimination?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}