{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3dbbb074-32cb-5863-8fe8-dc4443f40882","task_key":"default--test--5347","task_revision_id":"1","upstream_id":"5347","short_description":"If a CPA recklessly departs from the standards of due care when conducting an…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Unintentional tort.', 'Gross negligence.', 'Fraud.', 'Breach of contract.', 'Breach of warranty.', 'Negligence.', 'Misrepresentation.', 'Constructive fraud.', 'Professional malpractice.', 'Strict liability.']\",\"question\":\"If a CPA recklessly departs from the standards of due care when conducting an audit the CPA will be liable to third parties who are unknown to the CPA under the law of\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}