{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2331e897-377c-598e-96cc-2d3473b730f2","task_key":"default--test--6601","task_revision_id":"1","upstream_id":"6601","short_description":"A malignant tumour is damaging the patient's glossopharyngeal nerve. They will…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['loss of taste over the anterior two-thirds of the tongue.', 'complete loss of taste sensation.', 'paralysis of the muscles of the throat.', 'paralysis of the muscles of the jaw.', 'loss of somaesthetic sensation over the posterior two thirds of the tongue.', 'loss of somaesthetic sensation over the anterior two thirds of the tongue.', 'loss of taste over the posterior two-thirds of the tongue.', 'complete loss of somaesthetic sensation.', 'loss of taste and somaesthetic sensation over the posterior third of the tongue.', 'paralysis of the muscles of the tongue.']\",\"question\":\"A malignant tumour is damaging the patient's glossopharyngeal nerve. They will experience\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}