{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5d62aa41-c538-5fb8-9bb7-adcad0ed9fba","task_key":"default--test--2868","task_revision_id":"1","upstream_id":"2868","short_description":"The members of a bird species capable of nesting in treetops or on the ledges of…","config":"default","split":"test","body":"{\"category\":\"biology\",\"options\":\"['their potential niche only', 'their ecological niche only', 'their primary habitat only', 'both their fundamental niche and their realistic niche', 'their realistic niche only', 'their fundamental niche only', 'their secondary habitat only', 'their dominant niche only', 'their survival niche only', 'neither their fundamental niche nor their realistic niche']\",\"question\":\"The members of a bird species capable of nesting in treetops or on the ledges of nearby cliffs make their homes only in the treetops due to the established presence of a more aggressive bird species on the cliff ledges. With respect to habitat, the treetop-nesting birds are utilizing\",\"src\":\"ori_mmlu-college_biology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}