{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7fcdd0a-c6f2-54d9-a65b-42ca38bd2600","task_key":"default--test--2385","task_revision_id":"1","upstream_id":"2385","short_description":"Dr. Harry Holliday, a clinical psychologist, is planning to take a three-week…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Leave without informing his clients of his absence.', 'give his clients a phone number where he can be reached.', 'Provide his clients with online resources to manage their issues in his absence.', 'Continue his sessions via phone calls or video conferencing while on vacation.', 'Inform his clients only a day before his departure about his unavailability.', 'Cancel all appointments for the duration of his vacation without informing his clients.', 'discuss his vacation plans with his current clients ahead of time so that they know he’ll be unavailable during that time.', 'hire a colleague to take over his appointments while he is on vacation.', 'give his clients the phone number of a colleague who has agreed to accept phone calls in emergencies.', 'Arrange for a colleague to provide only emergency sessions, while regular appointments are paused.']\",\"question\":\"Dr. Harry Holliday, a clinical psychologist, is planning to take a three-week cruise. In terms of his clinical practice, Dr. Holliday’s best course of action would be to:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}