{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2de809e0-a1ba-537a-bad3-9c37e3e399c4","task_key":"default--test--2078","task_revision_id":"1","upstream_id":"2078","short_description":"You receive an e-mail from Dr. Brenda Browne, a licensed psychologist, who is…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Refuse to forward the file as it violates patient privacy laws.', 'wait until you receive a request from the client before taking any action.', 'Directly send the original file to the psychologist without any prior discussion with the client.', 'forward a photocopy of the file to the psychologist as requested.', \\\"Delete all the records related to the client's substance abuse disorder before sending the file.\\\", 'Send only the parts of the file that were created during your treatment of the client.', 'forward a summary of the file to the psychologist that includes only information related to the client’s current condition.', 'Ask Dr. Browne for a copy of the signed release before sending the file.', 'Forward the entire file digitally to the psychologist as requested.', 'contact the client to discuss the release of information.']\",\"question\":\"You receive an e-mail from Dr. Brenda Browne, a licensed psychologist, who is currently seeing one of your former clients for a substance abuse disorder. Dr. Browne asks you to forward the client’s file to her and states that the client has signed a release authorizing you to do so. You should:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}