{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f3e3f6d-9749-589b-a283-38d60d9bee13","task_key":"all--test--8f3e3f6d-9749-589b-a283-38d60d9bee13","task_revision_id":"1","upstream_id":"","short_description":"A pharmaceutical company wants to test a new cholesterol-reducing drug against…","config":"all","split":"test","body":"{\"choices\":\"['Block on gender because it is not associated with cholesterol level.', 'Block on gender because males and females are different.', 'Block on the type of drug because it may be associated with cholesterol level.', 'Block on exercise level because it is associated with cholesterol level.']\",\"question\":\"A pharmaceutical company wants to test a new cholesterol-reducing drug against the previous drug. It does not anticipate much association between cholesterol level and gender, but it does anticipate an association between cholesterol level and the amount of exercise a person gets. For a randomized block design, it should:\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}