{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c6bae49c-88c7-54e3-b213-d01925e15448","task_key":"all--test--c6bae49c-88c7-54e3-b213-d01925e15448","task_revision_id":"1","upstream_id":"","short_description":"Which of the following is a characteristic of the fault-tolerant nature of…","config":"all","split":"test","body":"{\"choices\":\"['The ability to use a hierarchical naming system to avoid naming conflicts', 'The ability to provide data transmission even when some connections have failed', 'The ability to resolve errors in domain name system (DNS) lookups', 'The ability to use multiple protocols such as hypertext transfer protocol (HTTP), Internet protocol (LP), and simple mail transfer protocol (SMTP) to transfer data']\",\"question\":\"Which of the following is a characteristic of the fault-tolerant nature of routing on the Internet?\",\"subject\":\"high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}