{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c2632c18-4759-50b6-8fa4-319fca4c1eb1","task_key":"default--test--1995","task_revision_id":"1","upstream_id":"1995","short_description":"An examinee who makes a number of confabulatory responses on the Rorschach has…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['an inability to perceive abstract concepts', 'high level of stress or anxiety', 'a low level of self-esteem', 'high intelligence and creativity', 'a lack of attention to detail', 'a need for immediate gratification', 'brain damage, intellectual disability, or emotional disturbance', 'a high degree of defensiveness or denial', 'a highly imaginative personality', 'a high degree of introversion or social withdrawal']\",\"question\":\"An examinee who makes a number of confabulatory responses on the Rorschach has overgeneralized from a part of the inkblot to the whole. Such responses suggest which of the following?\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}