{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9e8409c0-d551-5e6b-8452-79b7eb9f080a","task_key":"default--test--2782","task_revision_id":"1","upstream_id":"2782","short_description":"The Rosenhan study of mental institutions showed that","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['medical professionals are easily convinced when one claims to have a disorder.', 'patients at mental institutions often fake their symptoms.', 'treatment at private institutions tends to be better than treatment at public institutions.', \\\"confirmation bias may influence clinicians' views and treatments of mental patients.\\\", 'men are diagnosed at higher rates than women reporting the same symptoms.', 'mental institutions often misdiagnose physical ailments as mental disorders.', 'patients in mental institutions are usually treated based on their economic status.', 'women are diagnosed at higher rates than men reporting the same symptoms.', 'it is difficult to convince medical professionals that one has a disorder when one does not.', 'treatment at public institutions tends to be better than treatment at private institutions.']\",\"question\":\"The Rosenhan study of mental institutions showed that\",\"src\":\"ori_mmlu-high_school_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}