{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1e15f0f-7aea-584d-88de-67d8808a69eb","task_key":"default--test--2069","task_revision_id":"1","upstream_id":"2069","short_description":"Dr. Carl Carlson is a licensed psychologist and professor in the psychology…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['acting ethically as long as he discloses the relationship to his colleagues in the psychology department.', 'clearly violating ethical guidelines.', \\\"acting unethically only if their relationship affects Marla's academic performance.\\\", 'acting ethically as long as Marla is a student in a department other than the psychology department and will not be taking any classes in the psychology department.', 'acting ethically as long as he does not become sexually involved with Marla until she is no longer affiliated with the university.', 'acting ethically since he is not the one who initiated a potential relationship.', 'clearly violating ethical guidelines, unless he has received permission from the university.', 'acting ethically as long as he does not date any other students in the university.', 'acting unethically only if Marla is currently enrolled in one of his classes.', 'acting ethically as long as he maintains a professional relationship with Marla.']\",\"question\":\"Dr. Carl Carlson is a licensed psychologist and professor in the psychology department at a large university. He is approached by a graduate student, Marla M., age 36, who asks him if he’d like to attend a baseball game on Friday night. Dr. Carlson is divorced and finds Marla to be a very attractive woman. In terms of ethical guidelines, if Dr. Carlson accepts Marla’s invitation, he will be:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}