{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"59af5e5b-20b9-5dcd-9734-9a6f2be9e0df","task_key":"all--test--59af5e5b-20b9-5dcd-9734-9a6f2be9e0df","task_revision_id":"1","upstream_id":"","short_description":"This question refers to the following information.","config":"all","split":"test","body":"{\"choices\":\"['A puritanical injunction to adhere to a strict code of conduct', 'An ecumenical call for religious tolerance', 'A sermon about respecting political authority', 'A spirited attempt to convert non-believers']\",\"question\":\"This question refers to the following information.\\nO ye who believe! Strong drink and games of chance and idols… are only an infamy of Satan's handiwork. Leave it aside in order that ye may succeed.\\nthe Qur'an, Sura 5: 89\\nWhen once you hear the roses are in bloom,\\nThen is the time, my love, to pour the wine.\\nthe Rubáiyát of Omar Khayyám\\nThe tone of the first verse best reflects which of the following modes of religious discourse?\",\"subject\":\"high_school_world_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}