{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1e5579f-6292-5898-9315-211cead4c097","task_key":"all--test--d1e5579f-6292-5898-9315-211cead4c097","task_revision_id":"1","upstream_id":"","short_description":"Why do we know more about Neandertals than other extinct species of premodern…","config":"all","split":"test","body":"{\"choices\":\"['Neandertals spread throughout the world and their remains are found on every continent.', 'Neandertals used caves in Europe where their remains were preserved.', 'Neandertal populations were larger and more violent than any other premodern humans, whom they drove to extinction.', 'Neandertals left crude but intelligible written records.']\",\"question\":\"Why do we know more about Neandertals than other extinct species of premodern humans?\",\"subject\":\"prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}