{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bcb1b97d-bab0-59f2-b696-ee2ff4d4c7c5","task_key":"default--test--2759","task_revision_id":"1","upstream_id":"2759","short_description":"Wayne W. tells Dr. Pete Pannick that his partner is threatening to harm Dr.…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['should continue to see Wayne in therapy and ask Wayne to bring his partner to the next session to discuss the situation.', 'should continue seeing Wayne in therapy but, with his permission, contact the police to discuss the threat.', 'may terminate therapy with Wayne but only after providing him with pre-termination counseling and appropriate referrals.', 'may terminate therapy with Raymond.']\",\"question\":\"Wayne W. tells Dr. Pete Pannick that his partner is threatening to harm Dr. Panniclc because he's not happy with the advice that Wayne is getting in therapy. Dr. Panniclc knows that Wayne's partner has engaged in violent behavior in the past and believes the threat is serious. Dr. Panniclc:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}