{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9b72a1f9-5d9d-54b4-b592-55d28528d832","task_key":"default--test--5343","task_revision_id":"1","upstream_id":"5343","short_description":"A CPA prepared a tax return that involved a tax shelter transaction that was…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['There was substantial authority for the position.', 'The CPA had obtained written consent from the client for the position.', 'The tax shelter transaction was disclosed to the IRS separately.', 'There was a minimal chance of success for the position.', 'It is reasonable to believe that the position would more likely that not be upheld.', 'The CPA was not compensated for preparing the return.', 'The tax shelter transaction was a minor part of the return.', 'The CPA had no prior knowledge of the tax shelter transaction.', 'There was a reasonable basis for the position.', 'There was a reasonable possibility of success for the position.']\",\"question\":\"A CPA prepared a tax return that involved a tax shelter transaction that was disclosed on the return. In which of the following situations would a tax return preparer penalty not be applicable?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}