{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d4ea3c8-35aa-5bc1-aea2-2805ef6adfeb","task_key":"default--test--4948","task_revision_id":"1","upstream_id":"4948","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['The Consolation of Philosophy', 'Utopia', 'The Praise of Folly', 'The Decameron', 'The Divine Comedy', 'The Book of the Courtier', 'Oration on the Dignity of Man', 'The Prince', 'The Canterbury Tales', 'Handbook of the Christian Knight']\",\"question\":\"This question refers to the following information.\\n\\\"The substitution of Plato for the scholastic Aristotle was hastened by contact with Byzantine scholarship. Already at the Council of Ferrera (1438), which nominally reunited the Eastern and Western churches, there was a debate in which the Byzantines maintained the superiority of Plato to Aristotle. Cosimo and Lorenzo de Medici were both addicted to Plato; Cosimo founded and Lorenzo continued the Florentine Academy, which was largely devoted to the study of Plato&…The humanists of the time, however, were too busy acquiring knowledge of antiquity to be able to produce anything of value.\\\"\\nBertrand Russell, British philosopher, History of Western Philosophy, 1946\\nThe cultural diffusion described by Bertrand Russell most directly influenced the composition of which text?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}