{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d0a1f90f-687f-5e83-bb9e-3756046722aa","task_key":"all--test--d0a1f90f-687f-5e83-bb9e-3756046722aa","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['key economic processes within Islamic society.', 'production of food and drink in Islamic society.', 'religious regulation of personal behavior in Islamic society.', 'changing gender relations in Islamic society.']\",\"question\":\"This question refers to the following information.\\nO ye who believe! Strong drink and games of chance and idols… are only an infamy of Satan's handiwork. Leave it aside in order that ye may succeed.\\nthe Qur'an, Sura 5: 89\\nWhen once you hear the roses are in bloom,\\nThen is the time, my love, to pour the wine.\\nthe Rubáiyát of Omar Khayyám\\nBoth verses relate most closely to\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}