{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"875d1f34-0c37-5583-9305-7f9fd698cad1","task_key":"default--test--1986","task_revision_id":"1","upstream_id":"1986","short_description":"A person who received a Level 4 rating on the Rancho Los Amigos Scale of…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['is unresponsive to all stimuli but shows signs of basic physiological functioning such as breathing and heartbeat.', 'is highly responsive and coherent, but cannot remember or integrate past and recent events.', 'is functioning at an intellectual level that is average for his/her age, education, and demographic background.', 'is responsive to stimuli but cannot remember or integrate past and recent events, and exhibits impaired judgment.', 'is confused and incoherent, may exhibit bizarre behavior, and is unable to care for him/herself.', 'is functioning at an intellectual level that is superior for his/her age, education, and demographic background.', 'is nonresponsive to visual or auditory stimuli and seems to be in a state of deep sleep.', 'is alert and oriented and can remember and integrate remote and recent events but may have some impairment in judgment, planning, and abstract reasoning.']\",\"question\":\"A person who received a Level 4 rating on the Rancho Los Amigos Scale of Cognitive Functioning:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}