{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1f63435c-cf05-5a6f-8d80-c7f707d819fe","task_key":"default--test--10573","task_revision_id":"1","upstream_id":"10573","short_description":"A Chord based distributed hash table (DHT) with 26 address space is used in a…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['[11, 15, 23, 31, 40, 45]', '[1, 11, 15, 23, 31, 40]', '[12, 24, 36, 48, 60, 63]', '[3, 5, 7, 9, 11, 13]', '[2, 3, 5, 9, 17, 33]', '[7, 14, 28, 56, 63, 70]', '[4, 8, 16, 32, 48, 64]', '[13, 26, 39, 52, 65, 78]', '[2, 4, 6, 8, 10, 12]', '[5, 10, 20, 30, 40, 50]']\",\"question\":\"A Chord based distributed hash table (DHT) with 26 address space is used in a peer- to-peer file sharing network. There are currently 10 active peers in the network with node ID N1, N11, N15, N23, N31, N40, N45, N51, N60, and N63. Show all the target key (in ascending order, ignore the node's identifier itself) for N1.\",\"src\":\"theoremQA-EECS\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}