{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b5bee26d-87e3-5325-9f47-3f5e3ad81765","task_key":"all--test--b5bee26d-87e3-5325-9f47-3f5e3ad81765","task_revision_id":"1","upstream_id":"","short_description":"A large city was interested in annexing part of the surrounding county. In a…","config":"all","split":"test","body":"{\"choices\":\"['It is an example of nonresponse bias, the systematic tendency of individuals with particular characteristics to refuse to answer a survey question.', 'It is the systematic difference between a statistic and parameter caused by the nonrandom selection of surveyed persons.', 'It is the difference between the same statistics computed from two different samples.', 'It is the difference between the statistic and the truth due to use of a random sample.']\",\"question\":\"A large city was interested in annexing part of the surrounding county. In a survey conducted by the local newspaper, 58 percent of respondents said they were against the annexation. During the actual vote, not all eligible voters voted, but 56 percent of the respondents voted against the annexation. Which of the following best describes the difference in the percentages obtained from the newspaper poll and the vote itself?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}