{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"039ae08a-52cc-5fc8-82c0-42a386a71c2f","task_key":"default--test--10516","task_revision_id":"1","upstream_id":"10516","short_description":"In the NoNicks operating system, the time required by a single file-read…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['1:4', '1:2', '1:1', '1:1.5', '1:3.5', '3:1', '1.1:1', '2:1', '1.5:1', '1:2.5']\",\"question\":\"In the NoNicks operating system, the time required by a single file-read operation has four nonoverlapping components:\\ndisk seek time-25 msec\\ndisk latency time-8 msec\\ndisk transfer time- 1 msec per 1,000 bytes\\noperating system overhead-1 msec per 1,000 bytes + 10 msec\\nIn version 1 of the system, the file read retrieved blocks of 1,000 bytes. In version 2, the file read (along with the underlying layout on disk) was modified to retrieve blocks of 4,000 bytes. The ratio of-the time required to read a large file under version 2 to the time required to read the same large file under version 1 is approximately\",\"src\":\"ori_mmlu-college_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}