{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"985f28ce-4f17-5674-9e36-6f1eefb1173a","task_key":"default--test--6777","task_revision_id":"1","upstream_id":"6777","short_description":"Which one of the following represents a IIIrd cranial nerve palsy and is a…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Unilateral oval shaped pupil.', 'Bilateral oval shaped pupils.', 'Unilateral constricted and fixed pupil.', 'Unilateral constricted pupil.', 'Unilateral fixed dilated pupil.', 'Oval shaped pupils.', 'Bilateral fixed dilated pupils.', 'Bilateral constricted pupils.', 'Unilateral non-reactive pupil.', 'Bilateral non-reactive pupils.']\",\"question\":\"Which one of the following represents a IIIrd cranial nerve palsy and is a neurosurgical emergency?\",\"src\":\"ori_mmlu-clinical_knowledge\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}