{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4418ee8-f603-5556-a5da-3ed2a0c4ddeb","task_key":"all--test--c4418ee8-f603-5556-a5da-3ed2a0c4ddeb","task_revision_id":"1","upstream_id":"","short_description":"Two computers are built by different manufacturers. One is running a Web server…","config":"all","split":"test","body":"{\"choices\":\"['The computers cannot communicate because different manufacturers use different communication protocols.', \\\"The computers can communicate, but additional hardware is needed to convert data packets from one computer's protocol to the other computer's protocol.\\\", 'The computers can communicate directly only if the messages consist of text; other formats cannot be interpreted across computers.', 'The computers can communicate directly because Internet communication uses standard protocols.']\",\"question\":\"Two computers are built by different manufacturers. One is running a Web server and the other is running a Web browser. Which of the following best describes the ability of the two computers to communicate with each other across the Internet?\",\"subject\":\"high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}