{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"19a8e9f8-23a4-552e-8942-912e1b3bb40b","task_key":"all--test--19a8e9f8-23a4-552e-8942-912e1b3bb40b","task_revision_id":"1","upstream_id":"","short_description":"The tobacco industry has a lot of money, and it is expected to continue to do…","config":"all","split":"test","body":"{\"choices\":\"['More women start smoking this year than men who have quit.', 'People who continue to smoke consume more tobacco than in previous years.', 'The number of people who started smoking grass or taking snuff this year is more than the number who have given up smoking.', 'More cigarettes produced in the U.S. are exported to other countries this year than last year.']\",\"question\":\"The tobacco industry has a lot of money, and it is expected to continue to do so. Last year, the total amount of tobacco sold by American tobacco farmers increased significantly, but the number of smokers in the country continued to decline. If the following are true, they can be the reason for the above phenomenon, except:\",\"subject\":\"miscellaneous\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}