{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2fab269f-7ff1-522c-abfc-cd76581ed6a1","task_key":"default--test--1987","task_revision_id":"1","upstream_id":"1987","short_description":"You receive a phone call from Hermann H., age 28, who says he is “totally…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"[\\\"Discuss your discomfort with Hermann's political views in the first session.\\\", 'Ignore your personal feelings and continue therapy without discussing the difference in political views.', 'discuss the difference in political views with Hermann only if they become relevant to the psychotherapy process.', 'Suggest Hermann to find a psychologist who shares his political views.', \\\"Decline Hermann's request for therapy because of the difference in political views.\\\", 'see Hermann in therapy until his current crisis is over and then make a referral if necessary.', 'Tell Hermann outright that his political views are repugnant and continue the therapy.', 'Offer to treat Hermann only if he changes his political views.', 'provide Hermann with appropriate referrals.']\",\"question\":\"You receive a phone call from Hermann H., age 28, who says he is “totally miserable” because of the recent breakup with his girlfriend and that he would like to begin therapy with you. During the first session with Hermann, you find out that his political views are completely repugnant to you, and you feel that you would not enjoy working with him. As an ethical psychologist, you should:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}