{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f3016c3e-b23b-5423-b68f-d1d3cd45205f","task_key":"default--test--2438","task_revision_id":"1","upstream_id":"2438","short_description":"Whenever her clients cancel an appointment with Dr. Penny Pincher within less…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['typically legal, but considered unethical in certain jurisdictions.', 'illegal and unethical.', 'neither legal nor illegal, but ethical.', 'legal but unethical.', 'legal, but not considered best practice.', 'legal and ethical.', 'illegal, but considered best practice.', 'illegal but ethical.', 'neither legal nor illegal, but unethical.', 'illegal, but depends on the specific insurance policy.']\",\"question\":\"Whenever her clients cancel an appointment with Dr. Penny Pincher within less than 24 hours of the appointment, Dr. Pincher routinely bills the client’s insurance company for her full hourly fee. This practice is:\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}