{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c3c21cc8-d81b-5f83-86bd-782bd2983bfc","task_key":"all--test--c3c21cc8-d81b-5f83-86bd-782bd2983bfc","task_revision_id":"1","upstream_id":"","short_description":"In the NoNicks operating system, the time required by a single file-read…","config":"all","split":"test","body":"{\"choices\":\"['1:4', '1:3.5', '1:1', '1.1:1']\",\"question\":\"In the NoNicks operating system, the time required by a single file-read operation has four nonoverlapping components:\\ndisk seek time-25 msec\\ndisk latency time-8 msec\\ndisk transfer time- 1 msec per 1,000 bytes\\noperating system overhead-1 msec per 1,000 bytes + 10 msec\\nIn version 1 of the system, the file read retrieved blocks of 1,000 bytes. In version 2, the file read (along with the underlying layout on disk) was modified to retrieve blocks of 4,000 bytes. The ratio of-the time required to read a large file under version 2 to the time required to read the same large file under version 1 is approximately\",\"subject\":\"college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}