{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e8763e35-0dd7-59dc-b843-0252479ea787","task_key":"default--test--5597","task_revision_id":"1","upstream_id":"5597","short_description":"The population pyramid of a population that has had a slightly larger birth rate…","config":"default","split":"test","body":"{\"category\":\"other\",\"options\":\"['Narrow at the bottom and wide at the top', 'Having parallel sides', 'Having a diamond shape', 'Even width from bottom to top', 'Bulging at the top', 'Bulging in the middle', 'Wider at the top than in the middle', 'Wider at the bottom than at the top', 'Narrow at the top and wide in the middle', 'Narrow in the middle and wide at the top and bottom']\",\"question\":\"The population pyramid of a population that has had a slightly larger birth rate than death rate for several generations will most likely have which of the following shapes?\",\"src\":\"ori_mmlu-sociology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}