{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"87182733-46ac-5446-87a2-373b3dc10b3f","task_key":"default--test--10609","task_revision_id":"1","upstream_id":"10609","short_description":"In Chord, assume the size of the identifier space is 16. The active nodes are…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['[7, 10, 13, 14]', '[7, 9, 12, 15]', '[7, 8, 10, 14]', '[7, 8, 12, 14]', '[7, 9, 11, 13]', '[6, 8, 10, 14]', '[6, 7, 9, 11]', '[8, 9, 11, 13]', '[3, 6, 8, 12]', '[8, 10, 12, 14]']\",\"question\":\"In Chord, assume the size of the identifier space is 16. The active nodes are N3, N6, N8 and N12. Show all the target key (in ascending order, ignore the node's identifier itself) for N6.\",\"src\":\"theoremQA-EECS\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}