{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a44e0317-6e5b-591d-9e31-d1a741f8701d","task_key":"all--test--a44e0317-6e5b-591d-9e31-d1a741f8701d","task_revision_id":"1","upstream_id":"","short_description":"Suppose Unix did not provide a way of passing file descriptors between…","config":"all","split":"test","body":"{\"choices\":\"['True, True', 'False, False', 'True, False', 'False, True']\",\"question\":\"Suppose Unix did not provide a way of passing file descriptors between processes, but still allowed inheriting file descriptors from a parent on fork and exec. What aspects of the OKWS design would break without file descriptor passing?\\n1. It would be impossible for services to send messages to oklogd.\\n2. It would be impossible for services to get a TCP connection to a database proxy.\",\"subject\":\"computer_security\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}