{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07e23f37-6f6c-521f-ae0a-ae9c49108cbc","task_key":"all--test--07e23f37-6f6c-521f-ae0a-ae9c49108cbc","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['human beings have a soul', 'human beings know that they will die', 'it is possible for human beings to go to heaven', 'the potential of the human being is unlimited']\",\"question\":\"This question refers to the following information.\\nAt last it seems to me that I have come to understand why man is the most fortunate of all creatures and consequently worthy of all admiration. … The nature of all other beings is limited. … Imagine! The great generosity of God! The happiness of man! To man it is allowed to be whatever he chooses to be!\\nPico della Mirandola, Oration on the Dignity of Man, 1486\\nFrom the passage, one may infer that Pico believed that humankind was unique because\",\"subject\":\"high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}