{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae8956e6-f546-5510-aac4-b6476d038bad","task_key":"default--test--5265","task_revision_id":"1","upstream_id":"5265","short_description":"Which of the following is an element of a CPA firm's quality control policies…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Social media activity of the firm.', 'Personal relationships with clients.', 'Computer information processing.', 'Quality of office supplies.', 'Professional skepticism of management.', 'Public reputation of the firm.', 'Efficiency of organizational structures.', 'Time management strategies.', 'Acceptance of a client relationship.', 'Number of audit clients.']\",\"question\":\"Which of the following is an element of a CPA firm's quality control policies and procedures applicable to the firm's auditing practice?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}