{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4735f40d-bf73-58e2-b6cb-362d5d26212e","task_key":"all--test--4735f40d-bf73-58e2-b6cb-362d5d26212e","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['that man was a unique kind of creature', 'that God intended man to strive to achieve', 'that man is unworthy of admiration', \\\"that humans are God's equals\\\"]\",\"question\":\"This question refers to the following information.\\nAt last it seems to me that I have come to understand why man is the most fortunate of all creatures and consequently worthy of all admiration. … The nature of all other beings is limited. … Imagine! The great generosity of God! The happiness of man! To man it is allowed to be whatever he chooses to be!\\nPico della Mirandola, Oration on the Dignity of Man, 1486\\nFrom the passage, one may infer that Pico rejected the notion\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}