{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a914f7b8-cae8-56ec-a875-9299d7699342","task_key":"default--test--4934","task_revision_id":"1","upstream_id":"4934","short_description":"The commoners of the rural Aztec villages of Capilco and Cuexcomate:","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['had to pay a high tax in goods and labor to the local governor.', 'lived under constant threat of being sold into slavery.', 'were mostly left alone and did fairly well for themselves.', 'were under the direct control of the high priest, who demanded regular sacrifices.', \\\"were heavily policed by the king's guards to prevent any form of uprising.\\\", 'were ruled by an iron fist by the king, who had armies stationed there.', 'were forced to send half of all maize they grew to the king as tribute and lived in absolute poverty.', 'were required to send all their children to be educated in the city.', 'had no protection from city states that invaded from the north.', 'were forced to work on large-scale irrigation projects for the king.']\",\"question\":\"The commoners of the rural Aztec villages of Capilco and Cuexcomate:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}