{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"770de50a-d36e-5f21-9017-c6b03c5dfd75","task_key":"all--test--770de50a-d36e-5f21-9017-c6b03c5dfd75","task_revision_id":"1","upstream_id":"","short_description":"A talk show host recently reported that in response to his on-air question, 82%…","config":"all","split":"test","body":"{\"choices\":\"['The survey is meaningless because of voluntary response bias.', 'No meaningful conclusion is possible without knowing something more about the characteristics of his listeners.', 'The survey would have been more meaningful if he had picked a random sample of the 2500 listeners who responded.', 'The survey would have been more meaningful if he had used a control group.']\",\"question\":\"A talk show host recently reported that in response to his on-air question, 82% of the more than 2500 e-mail messages received through his publicized address supported the death penalty for anyone convicted of selling drugs to children. What does this show?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}