{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3775b2a8-5529-588a-bd3b-fc14f8167c54","task_key":"default--test--10560","task_revision_id":"1","upstream_id":"10560","short_description":"Statement 1| The BLEU metric uses precision, while the ROGUE metric uses recall.…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['Mostly True, Mostly False', 'Mostly False, Mostly True', 'False, True', 'False, Cannot be determined', 'Cannot be determined, False', 'True, True', 'Cannot be determined, True', 'False, False', 'True, Cannot be determined', 'True, False']\",\"question\":\"Statement 1| The BLEU metric uses precision, while the ROGUE metric uses recall. Statement 2| Hidden markov models were frequently used to model English sentences.\",\"src\":\"ori_mmlu-machine_learning\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}