{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b31ff241-d333-5d53-bf06-873b95e6605e","task_key":"all--test--b31ff241-d333-5d53-bf06-873b95e6605e","task_revision_id":"1","upstream_id":"","short_description":"A drug company will conduct a randomized controlled study on the effectiveness…","config":"all","split":"test","body":"{\"choices\":\"['Patients will spend more money on Heartaid, even though it is actually not any more effective than the current medication.', 'Doctors will begin to prescribe Heartaid to patients, even though it is actually not any more effective than the current medication.', 'Patients will continue to use the current medication, even though Heartaid is actually more effective.', 'Researchers will calculate the wrong P-value, making their advice to doctors invalid.']\",\"question\":\"A drug company will conduct a randomized controlled study on the effectiveness of a new heart disease medication called Heartaid. Heartaid is more expensive than the currently used medication. The analysis will include a significance test with H0: Heartaid and the current medication are equally effective at preventing heart disease and HA: Heartaid is more effective than the current medication at preventing heart disease. Which of these would be a potential consequence of a Type II error?\",\"subject\":\"high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}