{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae7d28c1-55bb-5031-ad44-4daa0d89769c","task_key":"default--test--2134","task_revision_id":"1","upstream_id":"2134","short_description":"A physician refers an asthmatic patient to you for therapy and requests regular…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"[\\\"agree to send reports, but only provide vague and generalized information about the patient's progress\\\", 'inform the physician that you can only send progress reports after the therapy is complete.', 'comply with this request, since a physician must be involved whenever you treat someone for a physical condition', 'refuse the request, stating that patient confidentiality prevents you from sharing any information', \\\"tell the physician that you will need the patient's family's written permission as well\\\", \\\"comply with this request, since the physician needs your input for best treatment of the patient's physical condition\\\", 'tell the physician that you can only provide verbal updates, not written reports', 'tell the physician chat psychological reports can be sent only to a psychiatrist or to another psychologist', 'comply with the request, but only send reports sporadically when you deem necessary', 'tell the physician thar you can send progress reports only with a written release by the patient']\",\"question\":\"A physician refers an asthmatic patient to you for therapy and requests regular progress reports. The most appropriate course of action for you to take is to\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}