{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fc8680ad-1406-5b65-af92-0d11dd2b9d2c","task_key":"default--test--8977","task_revision_id":"1","upstream_id":"8977","short_description":"The president of an online music streaming service whose customers pay a fee…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Choose all of the customers who joined in the last 6 months.', 'From the list of all customers who joined in the last 12 months, classify customers by the state in which they live, then choose 10 customers from each state.', 'Choose all of the customers who joined in the last month.', 'Choose a random sample of customers who joined in the last 12 months and have streamed at least 100 songs.', 'Make a list of all the customers who joined in the last 12 months and choose a random sample of customers on this list.', 'From the list of all customers who joined in the last 12 months, classify customers by the city in which they live, then choose 5 customers from each city.', 'Choose all of the customers who have joined in the last 12 months and have streamed at least 200 songs.', 'From the list of all customers who joined in the last 12 months, classify customers by the country in which they live, then choose 5% of the customers from each country.', 'From the list of all customers who joined in the last 12 months, classify customers by the genre of music they most frequently listen to, then choose a random sample from each genre.', 'From the list of all customers who joined in the last 12 months, classify customers by the state in which they live, then choose 3% of the customers from each state.']\",\"question\":\"The president of an online music streaming service whose customers pay a fee wants to gather additional information about customers who have joined in the past 12 months. The company plans to send out an e-mail survey to a sample of current customers with a link that gives participants a month of streaming service for free once the survey has been completed. They know that musical tastes vary by geographical region. Which of the following sample plans would produce the most representative sample of its customers?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}