{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6c9229b2-73a0-5943-8c90-92ab15d58de4","task_key":"default--test--10616","task_revision_id":"1","upstream_id":"10616","short_description":"DIMENSION A(10,10,10) Assuming that elements of the array are stored in…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['354, 987, 48', '456, 987, 84', '456, 987, 48', '456, 879, 48', '456, 798, 84', '345, 789, 84', '546, 987, 48', '345, 987, 48', '354, 798, 84', '546, 879, 84']\",\"question\":\"DIMENSION A(10,10,10) Assuming that elements of the array are stored in lexicographic order of their indices, find displacements of the locations of: (a) A(5,6,7)(b) A(10,9,8)(c) A(1,5,9) relative to the location of A(1,1,1).\",\"src\":\"stemez-ComputerScience\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}