{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ab58cf69-789f-5cf5-91cd-8978a97708cd","task_key":"all--test--ab58cf69-789f-5cf5-91cd-8978a97708cd","task_revision_id":"1","upstream_id":"","short_description":"The reason that archaeologists long suggested that the walls of Great Zimbabwe…","config":"all","split":"test","body":"{\"choices\":\"['the architecture was clearly built by Egyptians.', 'historical records show that it was constructed by early Dutch settlers.', 'symbolic motifs incorporated into the walls indicate that they were built by Indian traders.', 'they thought sub-Saharan Africans were incapable of such achievements.']\",\"question\":\"The reason that archaeologists long suggested that the walls of Great Zimbabwe could not have been built by an indigenous sub-Saharan people was that:\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}