{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4bf8518b-60cb-51ec-9885-2036b93be478","task_key":"default--test--2269","task_revision_id":"1","upstream_id":"2269","short_description":"Which of the following correctly completes the analogy: “Vision is to hearing as","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['parietal lobe is to frontal lobe.\\\"', 'temporal lobe is to parietal lobe.\\\"', 'temporal lobe is to frontal lobe.\\\"', 'occipital lobe is to parietal lobe.”', 'parietal lobe is to occipital lobe.\\\"', 'parietal lobe is to temporal lobe.”', 'occipital lobe is to temporal lobe.”', 'frontal lobe is to parietal lobe.\\\"', 'frontal lobe is to occipital lobe.”', 'occipital lobe is to frontal lobe.\\\"']\",\"question\":\"Which of the following correctly completes the analogy: “Vision is to hearing as\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}