{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3488a240-d55a-552d-8c44-2ba3bf4f155a","task_key":"default--test--10497","task_revision_id":"1","upstream_id":"10497","short_description":"Suppose there are three routers between a source host and a destination host.…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['[6, 3]', '[9, 4]', '[8, 4]', '[10, 5]', '[7, 4]', '[8, 3]', '[9, 5]', '[4, 2]', '[5, 3]', '[12, 6]']\",\"question\":\"Suppose there are three routers between a source host and a destination host. Ignoring fragmentation, an IP datagram sent from the source host to the destination host will travel over how many interfaces? How many forwarding tables will be indexed to move the datagram from the source to the destination? Answer in [Interfaces, Tables].\",\"src\":\"theoremQA-EECS\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}