{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28cc0ebc-3df6-523f-9311-58b8896b6679","task_key":"default--test--6304","task_revision_id":"1","upstream_id":"6304","short_description":"In the fetus, the ductus arteriosus passes blood from the","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['left ventricle to the aorta.', 'vena cava to the aorta.', 'pulmonary artery to the aorta.', 'pulmonary vein to the pulmonary artery.', 'aorta to the vena cava.', 'aorta to the pulmonary artery.', 'aorta to pulmonary vein.', 'pulmonary artery to the pulmonary vein.', 'aorta to the left ventricle.', 'pulmonary vein to the aorta.']\",\"question\":\"In the fetus, the ductus arteriosus passes blood from the\",\"src\":\"ori_mmlu-anatomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}