{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d57eec8-093b-5d98-8d13-3935d518ba30","task_key":"default--test--10569","task_revision_id":"1","upstream_id":"10569","short_description":"Suppose Unix did not provide a way of passing file descriptors between…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['False, True', 'False, False', 'True, True', 'True, False']\",\"question\":\"Suppose Unix did not provide a way of passing file descriptors between processes, but still allowed inheriting file descriptors from a parent on fork and exec. What aspects of the OKWS design would break without file descriptor passing?\\n1. It would be impossible for services to send messages to oklogd.\\n2. It would be impossible for services to get a TCP connection to a database proxy.\",\"src\":\"ori_mmlu-computer_security\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}