{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1e24db3-57e6-54f9-98e4-e67e2f895500","task_key":"default--test--2206","task_revision_id":"1","upstream_id":"2206","short_description":"A colleague, a licensed psychologist, confides in you that he has just ended a…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Advise the colleague to inform the Ethics Committee himself.', 'Contact the local authorities and report the colleague.', 'Advise the colleague to apologize to the client.', 'file a complaint against the colleague immediately with the Ethics Committee.', \\\"Ignore the matter as it's not your responsibility.\\\", 'Inform the Ethics Committee without notifying the colleague.', 'contact the client and discuss the matter with her.', 'Discuss the matter with other colleagues to seek their advice.', 'discuss the matter further with the colleague to see what course of action he plans to take.']\",\"question\":\"A colleague, a licensed psychologist, confides in you that he has just ended a sexual relationship with one of his clients. As he describes the client, you realize that she is someone you referred to him about 18 months ago. As an ethical psychologist, you should:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}