{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"779027dd-4ff9-5f9b-8b05-856720d65445","task_key":"default--test--2435","task_revision_id":"1","upstream_id":"2435","short_description":"You receive a call from Mrs. Wang who is very upset because her 14-year-old…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['see Mrs. Wang’s daughter in therapy only until the crisis has ended or until you locate alternative services.', 'Offer to train Mrs. Wang in providing crisis intervention to her daughter.', \\\"Suggest that Mrs. Wang's daughter should wait for a few days until she feels better before starting therapy.\\\", 'Agree to see Mrs. Wang’s daughter in therapy without informing her of your lack of experience.', 'inform Mrs. Wang that you cannot see her daughter because of your lack of experience.', 'Suggest that Mrs. Wang’s daughter should try to forget about the incident.', \\\"Recommend Mrs. Wang's daughter for group therapy with other adolescents.\\\", 'agree to see Mrs. Wang’s daughter in therapy since you’ve had some experience providing crisis intervention experiences.']\",\"question\":\"You receive a call from Mrs. Wang who is very upset because her 14-year-old daughter witnessed the assault of her best friend three days ago and is very distraught. The girl hasn’t slept, won’t eat, and can’t stop crying. You have limited experience working with adolescents and in providing crisis intervention services. However, there is no one else in the community who is more experienced than you are. As an ethical psychologist, you will:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}