{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a0df8243-7dea-5762-837d-8e2484f72d47","task_key":"all--test--a0df8243-7dea-5762-837d-8e2484f72d47","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['By imposing a universal currency based on the French franc', 'By the brutal suppression of guerrilla resistance', 'By abolishing feudalism and manorialism', 'By encouraging the use of French as the universal language']\",\"question\":\"This question refers to the following information.\\nRead the following excerpt.\\nNever were talents of the highest genius of the most exalted kind, more profusely bestowed upon a human being. The genius of Napoleon is astounding. All branches of human knowledge seemed alike familiar to his gigantic mind. His conversations at St. Helena, scattered through the numerous and voluminous memorials of those who gleaned them, are replete with intensest interest. During the long agony of his imprisonment and his death, he conversed with perfect freedom upon the events of his marvelous career, and upon all those subjects or morals, politics, and religion, which most deeply concern the welfare of our race. There is no mind which will not be invigorated by familiarity with these profound thoughts, expressed with so much glow of feeling and energy of diction.\\n—John S. C. Abbott, historian, Napoleon at St. Helena, 1855\\nNapoleon helped make the French Revolution an international movement in the areas he conquered\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}