{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3b5d6dfe-a18e-5e54-b1fb-be9fee7f90cb","task_key":"default--test--8447","task_revision_id":"1","upstream_id":"8447","short_description":"To determine the average number of children living in single-family homes, a…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Too high, because the sample size is not large enough.', 'Too low, because nonresponse bias tends to underestimate average results.', 'Too high, because convenience samples overestimate average results.', 'Too low, because of undercoverage bias.', 'Too low, because the sample size is not large enough.', 'Too high, due to the possibility of a skewed sample.', 'Too high, because of undercoverage bias.', 'Too low, because convenience samples overestimate average results.', 'Too low, due to the possibility of a skewed sample.', 'Too high, because nonresponse bias tends to overestimate average results.']\",\"question\":\"To determine the average number of children living in single-family homes, a researcher picks a simple random sample of 50 such homes. However, even after one follow-up visit the interviewer is unable to make contact with anyone in 8 of these homes. Concerned about nonresponse bias, the researcher picks another simple random sample and instructs the interviewer to keep trying until contact is made with someone in a total of 50 homes. The average number of children is determined to be 1.73. Is this estimate probably too low or too high?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}