{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a10804cf-8248-5aeb-8162-ea3c8796c29f","task_key":"default--test--10391","task_revision_id":"1","upstream_id":"10391","short_description":"Statement 1| Highway networks were introduced after ResNets and eschew max…","config":"default","split":"test","body":"{\"category\":\"computer science\",\"options\":\"['True, True', 'Both statements are partially true', 'The first statement is false, the second is partially true', 'The first statement is partially true, the second is false', 'The first statement is true, the second is undetermined', 'False, False', 'True, False', 'Both statements are undetermined', 'The first statement is undetermined, the second is true', 'False, True']\",\"question\":\"Statement 1| Highway networks were introduced after ResNets and eschew max pooling in favor of convolutions. Statement 2| DenseNets usually cost more memory than ResNets.\",\"src\":\"ori_mmlu-machine_learning\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}