{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"56ae2087-9e15-59da-b336-5a20b1bcf94e","task_key":"default--test--8982","task_revision_id":"1","upstream_id":"8982","short_description":"A kidney dialysis center periodically checks a sample of its equipment and…","config":"default","split":"test","body":"{\"category\":\"math\",\"options\":\"['Dialysis center: Type II error, towel manufacturer: Type II error', 'Dialysis center: Neither Type I nor Type II error, towel manufacturer: Type II error', 'Dialysis center: Type I error, towel manufacturer: Type II error', 'Dialysis center: Type II error, towel manufacturer: Type I error', 'Dialysis center: Both Type I and Type II errors, towel manufacturer: Type I error', 'Dialysis center: Type I error, towel manufacturer: Type I error', 'Dialysis center: Neither Type I nor Type II error, towel manufacturer: Type I error', 'Dialysis center: Both Type I and Type II errors, towel manufacturer: Type II error', 'Dialysis center: Type I error, towel manufacturer: Neither Type I nor Type II error', 'Dialysis center: Type II error, towel manufacturer: Neither Type I nor Type II error']\",\"question\":\"A kidney dialysis center periodically checks a sample of its equipment and performs a major recalibration if readings are sufficiently off target. Similarly, a fabric factory periodically checks the sizes of towels coming off an assembly line and halts production if measurements are sufficiently off target. In both situations, we have the null hypothesis that the equipment is performing satisfactorily. For each situation, which is the more serious concern, a Type I or Type II error?\",\"src\":\"ori_mmlu-high_school_statistics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}