{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b90887bd-82cc-53d8-9f07-87c7652aaf77","task_key":"all--test--b90887bd-82cc-53d8-9f07-87c7652aaf77","task_revision_id":"1","upstream_id":"","short_description":"\"I counted seven ministers all preaching at once . . . . Some of the people were…","config":"all","split":"test","body":"{\"choices\":\"['A meeting of the Pennsylvania Quakers during the time of the Revolution', 'A frontier camp meeting of the Second Great Awakening during the early nineteenth century', 'A presentation to a Chautauqua Assembly during the late nineteenth century', \\\"A convention of the Women's Christian Temperance Union during the late nineteenth century\\\"]\",\"question\":\"\\\"I counted seven ministers all preaching at once . . . . Some of the people were singing, others praying, some crying for mercy in the most piteous accents.\\\" The quotation above describes group participation in an event that was most characteristic of which of the following?\",\"subject\":\"miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}