{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2b81c6ac-dc17-55fe-b3c2-52768bd5041d","task_key":"default--test--5042","task_revision_id":"1","upstream_id":"5042","short_description":"Archaeological evidence for the collapse of civilizations suggests which of the…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the existence of a written language', 'the number of years the civilization has existed', \\\"the civilization's religious beliefs and practices\\\", 'whether warfare can be ended', \\\"the size of the civilization's population\\\", 'the ability of bureaucracies to store food and feed its people', \\\"the civilization's trade networks with other societies\\\", 'changing environmental conditions', 'the development of new technologies', 'how societies respond to challenges']\",\"question\":\"Archaeological evidence for the collapse of civilizations suggests which of the following is the most important variable?\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}