{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b366aff3-154e-5537-a938-f4d9ed69bb47","task_key":"default--test--2291","task_revision_id":"1","upstream_id":"2291","short_description":"Dr. Ramchandran found that his patients who brushed their teeth after lunch had…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"[\\\"The writer's statement would be justified if at least half of the patients brushed their teeth after lunch.\\\", \\\"If a minimum of 200 patients were studied, the writer's statement is justified.\\\", \\\"No matter how many participants, the writer's statement is not justified.\\\", \\\"The writer's statement would only be justified if no patients brushed their teeth after lunch.\\\", \\\"The writer's statement could only be justified if the study was conducted over a prolonged period of time.\\\", \\\"At least 100 of the patients needed to have brushed their teeth after lunch for the writer's statement to be justified.\\\", \\\"If all the patients were studied, the writer's statement is justified.\\\", \\\"If at least 100 patients were studied, the writer's statement is justified.\\\", \\\"If only the patients who brushed their teeth after lunch were studied, the writer's statement is justified.\\\", \\\"If a minimum of 500 patients were studied, the writer's statement is justified.\\\"]\",\"question\":\"Dr. Ramchandran found that his patients who brushed their teeth after lunch had 1/20 the number of cavities in their teeth as those who didn't. After interviewing the dentist, a local newswriter reports that brushing teeth after lunch prevents cavities. Based on the dentist's research, which of the following statements is true?\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}