{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93959025-a3e5-5a23-9dee-5fdaeafd685f","task_key":"all--test--93959025-a3e5-5a23-9dee-5fdaeafd685f","task_revision_id":"1","upstream_id":"","short_description":"In response to a column printed in newspapers throughout the United States, an…","config":"all","split":"test","body":"{\"choices\":\"['Her participants were not randomly assigned in her study.', 'The number of respondents from across the country was too small.', 'The study was not replicated.', 'Her sample may not have been representative of the population.']\",\"question\":\"In response to a column printed in newspapers throughout the United States, an advice columnist received over 28,000 responses. Over 75% of respondents said that if they had it to do over again, they would not have children. The columnist concludes that most parents are sorry that they had children. For which of the following reasons is her conclusion not valid?\",\"subject\":\"high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}