{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2031d393-afd1-5092-b684-ca38d461cf07","task_key":"all--test--2031d393-afd1-5092-b684-ca38d461cf07","task_revision_id":"1","upstream_id":"","short_description":"As the single-person public relations staff of a public transit agency, you are…","config":"all","split":"test","body":"{\"choices\":\"['Consider the comprehension level of all of the target audiences.', 'Consider how riding the bus could similarly affect all of the target audiences.', 'Consider which mediums would most effectively reach all of the target audiences.', 'Consider the geographic locations where all of the target audiences live and work.']\",\"question\":\"As the single-person public relations staff of a public transit agency, you are tasked with increasing the number of people who ride your buses each month. Your target audiences are lower-income individuals, college students, people with basic educations and people for whom English is a Second Language. When preparing to craft a primary message, what should you do first?\",\"subject\":\"public_relations\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}