{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"faa87daa-ec49-5d3b-904c-40c1b2dde844","task_key":"all--test--faa87daa-ec49-5d3b-904c-40c1b2dde844","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['Many Muslims inclined toward atheism in the premodern era.', 'Religious restrictions were not universally observed by all Muslims.', 'Christian missionaries were steadily winning converts among Muslim populations.', 'Muslim intellectuals were generally hostile to traditional religious practice.']\",\"question\":\"This question refers to the following information.\\nO ye who believe! Strong drink and games of chance and idols… are only an infamy of Satan's handiwork. Leave it aside in order that ye may succeed.\\nthe Qur'an, Sura 5: 89\\nWhen once you hear the roses are in bloom,\\nThen is the time, my love, to pour the wine.\\nthe Rubáiyát of Omar Khayyám\\nA comparison of both verses would best support which of the following conclusions?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}