{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cf1849b9-b9b7-501b-8feb-96417d4c77c5","task_key":"default--test--8335","task_revision_id":"1","upstream_id":"8335","short_description":"A large city was interested in annexing part of the surrounding county. In a…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['It is the difference caused by the undercoverage of certain groups in the population.', 'It is the difference between the statistic and the truth due to use of a random sample.', 'It is the systematic difference between a statistic and parameter caused by a biased question.', 'It is an example of response bias, the systematic tendency of individuals with particular characteristics to respond to a survey question in a certain way.', 'It is the difference caused by the overrepresentation of certain groups in the population.', 'It is the difference between the same statistics computed from two identical samples.', 'It is the systematic difference between a statistic and parameter caused by the nonrandom selection of surveyed persons.', 'It is the difference between the same statistics computed from two different samples.', 'It is an example of nonresponse bias, the systematic tendency of individuals with particular characteristics to refuse to answer a survey question.']\",\"question\":\"A large city was interested in annexing part of the surrounding county. In a survey conducted by the local newspaper, 58 percent of respondents said they were against the annexation. During the actual vote, not all eligible voters voted, but 56 percent of the respondents voted against the annexation. Which of the following best describes the difference in the percentages obtained from the newspaper poll and the vote itself?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}