{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1be99c32-12a3-51a2-85ab-bd0299606e49","task_key":"default--test--5637","task_revision_id":"1","upstream_id":"5637","short_description":"An accountant has been engaged to compile a nonissuer's financial statements…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Issue a standard compilation report without any modifications.', 'Withdraw from the compilation engagement and provide no further services concerning these financial statements.', 'Advise the management to hire another accountant for the engagement.', 'Inform the regulatory authorities about the misapplications of accounting principles.', \\\"Inform management that the engagement can proceed only if distribution of the accountant's compilation report is restricted to internal use.\\\", 'Continue the engagement and issue a modified report without informing the board of directors.', 'Disclaim an opinion on the financial statements and advise the board of directors that the financial statements should not be relied upon.', 'Proceed with the engagement but add a disclaimer in the report that the accountant is not responsible for the misapplications.', 'Proceed with the engagement but refuse to sign the compilation report.', 'Determine the effects of the deficiencies and add a separate paragraph to the compilation report that describes the deficiencies and their effects.']\",\"question\":\"An accountant has been engaged to compile a nonissuer's financial statements that contain several misapplications of accounting principles and unreasonable accounting estimates. Management is unwilling to revise the financial statements and the accountant believes that modification of the standard compilation report is not adequate to communicate the deficiencies. Under these circumstances the accountant should\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}