{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d85e9904-25e7-5c45-8f43-99215b626f11","task_key":"all--test--d85e9904-25e7-5c45-8f43-99215b626f11","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Civil wars between rivals for the throne', 'Incursions of barbarian peoples into the empire', 'The persistence of the senate as a body of government in the empire', 'Demographic weakness on account of diseases and epidemics']\",\"question\":\"This question refers to the following information.\\n\\\"After the Tencteri came, in former days, the Bructeri; but the general account now is, that the Chamavi and Angrivarii entered their settlements, drove them out and utterly exterminated them with the common help of the [neighboring] tribes, either from hatred of their tyranny, or from the attractions of plunder, or from heaven's [favorable] regard for us. It did not even grudge us the spectacle of the conflict. More than sixty thousand fell, not beneath the Roman arms and weapons, but, grander far, before our delighted eyes. May the tribes, I pray, ever retain if not love for us, at least hatred for each other; for while the destinies of empire hurry us on, fortune can give no greater boon than discord among our foes.\\\"\\nGermania, Publius Cornelius Tacitus, ca. 98 C.E.\\nWhich of the following contributed LEAST to the decline of the Roman Empire?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}