{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"09a1d58d-55e8-51f2-b8be-ad0df5d93de7","task_key":"default--test--2448","task_revision_id":"1","upstream_id":"2448","short_description":"You are subpoenaed to appear at a deposition to testify about a former client of…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['provide the information only if it puts the former client in a favorable light.', 'provide the information if the client has given verbal permission.', 'provide the information if you believe it is pertinent to the case.', \\\"provide the information if the opposing party's attorney insists.\\\", 'refuse to provide the information unless the client is present.', 'disclose the information if it has already been a reasonable time since the client-attorney relationship ended.', 'provide the information since privilege is waived in this situation.', \\\"provide the information if it doesn't harm the client's case.\\\", 'assert the privilege unless the client has signed a waiver.', 'refuse to provide the information without a court order.']\",\"question\":\"You are subpoenaed to appear at a deposition to testify about a former client of yours. During the deposition, the attorney for the opposing party asks you to provide confidential client information. You should:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}