{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bde99211-a62c-5e34-91ba-a2c3a7771d55","task_key":"default--test--2392","task_revision_id":"1","upstream_id":"2392","short_description":"Which of the following is the proper comparison rate for studying the…","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['Recovery rate in individuals using self-help resources', 'Incidence in the population of the disorder being eared', 'Recovery rate in patients using alternative therapies', 'Improvement rate in patients receiving no therapy', 'Cure rate for older, outdated treatment methods', 'Cure rate for the currently most effective therapists', 'Cure rate for treatments considered less effective', 'Relapse in those who experience spontaneous remission', 'Cure rare for current treatment modalities', 'Incidence of the disorder in the general population not receiving the new psychotherapy']\",\"question\":\"Which of the following is the proper comparison rate for studying the effectiveness of a new psychotherapy\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}