{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"28885ec8-a037-52ec-b151-0329f74b09c7","task_key":"default--test--2265","task_revision_id":"1","upstream_id":"2265","short_description":"A psychologist completes an assessment, then refers the patient to another…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['ethical only if the patient, not the therapist, pays for the consultation', 'unethical', 'unethical only if the therapist uses the consultation to diagnose the patient', 'unethical only if the therapist does not inform the patient about the consultation', 'ethical only if the consultation is free of charge', \\\"unethical only if the referring psychologist does not have the patient's consent\\\", 'ethical', 'ethical only if the therapist shares the consultation details with the patient', 'ethical only if the therapist is being supervised by the referring psychologist', 'ethical only if the referring psychologist and the therapist have a pre-existing professional relationship']\",\"question\":\"A psychologist completes an assessment, then refers the patient to another psychologist for therapy. The therapist requests and receives a consultation from the referring psychologist regarding the assessment, and pays for the consultation. This action is\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}