{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fda3ca8a-a8eb-5746-b5c1-1981b8769b68","task_key":"default--test--8449","task_revision_id":"1","upstream_id":"8449","short_description":"A drug company will conduct a randomized controlled study on the effectiveness…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Patients will suffer from side effects of Heartaid, even though it is not any more effective than the current medication.', 'The drug company will continue to produce Heartaid, even though it is not any more effective than the current medication.', 'Doctors will stop prescribing the current medication, even though Heartaid is not any more effective.', 'Researchers will incorrectly reject the null hypothesis, leading to a false conclusion about the effectiveness of Heartaid.', 'Researchers will calculate the wrong P-value, making their advice to doctors invalid.', 'The drug company will lose money because Heartaid is actually not any more effective than the current medication.', 'Patients will continue to use the current medication, even though Heartaid is actually more effective.', 'Doctors will begin to prescribe Heartaid to patients, even though it is actually not any more effective than the current medication.', 'The FDA will approve Heartaid for use, even though it is not any more effective than the current medication.', 'Patients will spend more money on Heartaid, even though it is actually not any more effective than the current medication.']\",\"question\":\"A drug company will conduct a randomized controlled study on the effectiveness of a new heart disease medication called Heartaid. Heartaid is more expensive than the currently used medication. The analysis will include a significance test with H0: Heartaid and the current medication are equally effective at preventing heart disease and HA: Heartaid is more effective than the current medication at preventing heart disease. Which of these would be a potential consequence of a Type II error?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}