{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"56275a1b-b41b-5e0b-9136-6e81f615b719","task_key":"all--test--56275a1b-b41b-5e0b-9136-6e81f615b719","task_revision_id":"1","upstream_id":"","short_description":"An insurance company wishes to study the number of years drivers in a large city…","config":"all","split":"test","body":"{\"choices\":\"['A reasonable time-and-cost-saving procedure would be to use systematic sampling on an available list of all AAA (Automobile Association of America) members in the city.', 'A reasonable time-and-cost-saving procedure would be to randomly choose families and include all drivers in each of these families in the sample.', 'To determine the mean number of years between accidents, randomness in choosing a sample of drivers is not important as long as the sample size is very large.', 'None of the above are true statements.']\",\"question\":\"An insurance company wishes to study the number of years drivers in a large city go between automobile accidents. They plan to obtain and analyze the data from a sample of drivers. Which of the following is a true statement?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}