{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"94165def-9a19-5037-827c-209879ac704f","task_key":"default--test--5108","task_revision_id":"1","upstream_id":"5108","short_description":"The tobacco industry has a lot of money, and it is expected to continue to do…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['There has been an increase in the production of smokeless tobacco products.', 'More people are using tobacco for medicinal purposes.', 'More women start smoking this year than men who have quit.', 'More cigarettes produced in the U.S. are exported to other countries this year than last year.', 'The number of people smoking cigars has increased this year.', 'People who continue to smoke consume more tobacco than in previous years.', 'The number of people who smoke occasionally has increased.']\",\"question\":\"The tobacco industry has a lot of money, and it is expected to continue to do so. Last year, the total amount of tobacco sold by American tobacco farmers increased significantly, but the number of smokers in the country continued to decline. If the following are true, they can be the reason for the above phenomenon, except:\",\"src\":\"ori_mmlu-miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}