{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6a51cd35-9af0-5045-beaa-d8bfaa13ac9e","task_key":"default--test--5126","task_revision_id":"1","upstream_id":"5126","short_description":"While performing certain nonaudit services for an insurance company, a…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Agree with the accounting treatment recommended by the company’s hedge fund trader.', 'Refuse to conduct the research and make a recommendation, because of a conflict of interest.', 'Refuse to conduct the research and make a recommendation, because of insufficient experience.', 'Agree to recommend the appropriate accounting treatment without performing any research on weather hedges.', 'Agree to conduct the research but refuse to make a recommendation, due to a lack of experience.', 'Agree to recommend the appropriate accounting treatment after performing sufficient research on weather hedges.', 'Agree to recommend the accounting treatment suggested by a third-party consultant without verification.', \\\"Refuse to conduct the research and make a recommendation, because it is outside of the accountant's area of expertise.\\\", 'Agree to conduct the research and make a recommendation, despite the potential conflict of interest.']\",\"question\":\"While performing certain nonaudit services for an insurance company, a professional accountant is asked to recommend the appropriate accounting treatment for a weather hedge transaction. The accountant has worked with financial hedges but has no experience with weather hedges. Which of the following actions by the accountant would be in compliance with the IFAC Code of Ethics for Professional Accountants?\",\"src\":\"ori_mmlu-professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}