{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fd04ea34-13d4-506e-a58b-15e2acb6ad53","task_key":"default--test--2066","task_revision_id":"1","upstream_id":"2066","short_description":"A father brings his 22-year-old son to you for evaluation. Within the past 6…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Schizophrenia', 'Paranoid Personality Disorder', 'Brief Reactive Psychosis', 'Bipolar Disorder', 'Schizophreniform Disorder', 'Antisocial Personality Disorder', 'Dissociative Identity Disorder', 'Obsessive-Compulsive Disorder', 'Generalized Anxiety Disorder', 'Major Depressive Disorder']\",\"question\":\"A father brings his 22-year-old son to you for evaluation. Within the past 6 weeks, this formerly socially active young man has quit his job as a restaurant manager, has become withdrawn, and now spends most of his time in his room watching TV. He reports that he occasionally receives messages from aliens through the TV. Based on this information, which of the following is the most appropriate DSM-IV diagnosis\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}