{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"983b316f-2d15-556e-9df8-26f625741725","task_key":"default--test--5272","task_revision_id":"1","upstream_id":"5272","short_description":"A CPA prepares income tax returns for a client. After the client signs and mails…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Promptly resign from the engagement and cooperate with the successor accountant.', 'Prepare an amended return immediately and send it to the client for signature.', \\\"Immediately inform the client's legal counsel about the error.\\\", 'Prepare an amended return within 30 days of the discovery of the error.', 'Promptly advise the client of the error.', 'Take responsibility for the error and pay any additional taxes owed due to the mistake.', 'Document the error in the workpapers.', 'Charge the client an additional fee for discovering the error.', 'Inform the IRS about the error directly without notifying the client.', 'Ignore the error since the return has already been signed and mailed.']\",\"question\":\"A CPA prepares income tax returns for a client. After the client signs and mails the returns, the CPA discovers an error. According to Treasury Circular 230, the CPA must\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}