{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8df774d8-0cb8-5953-9b0f-668ceca53157","task_key":"default--test--10649","task_revision_id":"1","upstream_id":"10649","short_description":"Of the following potential benefits, which is LEAST likely to be provided by the…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Customers will receive instant solutions to their issues.', 'Customers are likely to spend less time listening to information not relevant to their issue.', 'Human representatives will not be needed to respond to some inquiries.', 'Customers will be unable to mistakenly select the incorrect department for their particular issue.', 'The system will automatically resolve all customer issues.', 'The company will be able to provide a human representative for any incoming call.']\",\"question\":\"Of the following potential benefits, which is LEAST likely to be provided by the upgraded system?\",\"src\":\"ori_mmlu-high_school_computer_science\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}