{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ff5c2e38-3000-5701-ab96-11593cfb3794","task_key":"default--test--6085","task_revision_id":"1","upstream_id":"6085","short_description":"Blood clots are responsible for repeatedly blocking a catheter. What should you…","config":"default","split":"test","body":"{\"category\":\"health\",\"options\":\"['Use a syringe to remove the clot.', 'Remove the catheter and recatheterize.', 'Replace the catheter with a new one without informing medical staff.', \\\"Inform the patient's family.\\\", 'Inform the patient.', 'Ignore it and continue with the procedure.', 'Flush the catheter with saline.', 'Inform medical staff.', 'Repeatedly press the catheter to remove the clot.']\",\"question\":\"Blood clots are responsible for repeatedly blocking a catheter. What should you do?\",\"src\":\"ori_mmlu-clinical_knowledge\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}