{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b317c910-9e6f-5f61-8c65-bc50c3b7f824","task_key":"default--test--5127","task_revision_id":"1","upstream_id":"5127","short_description":"A CPA has been requested by a former audit client to reissue the auditor's…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"[\\\"Assess the professional reputation of the former client's management.\\\", 'Request permission from the current auditor to reissue the report.', \\\"Obtain a letter of representation from the former client's management.\\\", \\\"Review the former client's current financial statements.\\\", \\\"Check the former client's credit history.\\\", \\\"Consult with the former client's shareholders before reissuing the report.\\\", \\\"Evaluate the former client's overall financial stability.\\\", \\\"Review the former client's records to verify its compliance with debt and loan agreements.\\\"]\",\"question\":\"A CPA has been requested by a former audit client to reissue the auditor's report for the prior period. Before reissuing the report, the CPA should\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}