{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f7ad4e48-01bb-57af-bdb2-048e7bf7ab55","task_key":"all--test--f7ad4e48-01bb-57af-bdb2-048e7bf7ab55","task_revision_id":"1","upstream_id":"","short_description":"The president of an online music streaming service whose customers pay a fee…","config":"all","split":"test","body":"{\"choices\":\"['Choose all of the customers who joined in the last month.', 'Make a list of all the customers who joined in the last 12 months and choose a random sample of customers on this list.', 'From the list of all customers who joined in the last 12 months, classify customers by the state in which they live, then choose 10 customers from each state.', 'From the list of all customers who joined in the last 12 months, classify customers by the state in which they live, then choose 3% of the customers from each state.']\",\"question\":\"The president of an online music streaming service whose customers pay a fee wants to gather additional information about customers who have joined in the past 12 months. The company plans to send out an e-mail survey to a sample of current customers with a link that gives participants a month of streaming service for free once the survey has been completed. They know that musical tastes vary by geographical region. Which of the following sample plans would produce the most representative sample of its customers?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}