{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f684e702-bd94-5fe0-bc9b-65772c4a1f01","task_key":"default--test--4736","task_revision_id":"1","upstream_id":"4736","short_description":"The fact that early excavators of Teotihuacán mistook ordinary housing compounds…","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the society was a theocracy, ruled by high priests.', 'commoners took over the residences of the elite.', 'the city was a democracy, ruled by common farmers and artisans.', 'the general standard of living was high.', 'the elite lived in ordinary housing compounds to blend in with the commoners.', 'the housing compounds were built with materials typically used for palaces.', \\\"the city's economic system was based on wealth distribution.\\\", 'the city was abandoned and later occupied by a different civilization.', 'the society was mostly egalitarian, with almost no social stratification.', 'the excavators were inexperienced in identifying historical structures.']\",\"question\":\"The fact that early excavators of Teotihuacán mistook ordinary housing compounds for palaces suggests:\",\"src\":\"ori_mmlu-prehistory\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}