{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"138d1d95-eabf-5cd7-814a-d7ce26bb949b","task_key":"default--test--5003","task_revision_id":"1","upstream_id":"5003","short_description":"The Egyptian system of hieroglyphics:","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['was a simple form of writing with one symbol representing one word', 'all of the above', 'appears to have developed suddenly', 'did not use pictographs', 'was borrowed from the Sumerians', 'was only deciphered in the 20th century', 'was only used for religious texts', 'was replaced by the Latin alphabet during the Roman period', 'was the earliest form of writing in the world', 'was only used by the royal family']\",\"question\":\"The Egyptian system of hieroglyphics:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}