{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"71320b7b-c071-56c3-abe8-4ec984ca803d","task_key":"default--test--4852","task_revision_id":"1","upstream_id":"4852","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Many believed in the necessity of preserving the traditional religious doctrines.', 'Most rejected religion altogether and adopted atheism as the only credo of a rational man.', 'They were largely disconnected from the realities of the common people.', 'They endorsed the absolute power of the monarchies and aristocracies.', 'They believed that human suffering and disasters are inevitable and justified.', 'They rejected the idea of scientific discoveries having any impact on social and cultural issues.', 'Most actively advocated for the abolition of all forms of governance.', 'They utilized new methods of communicating their ideas, such as salons and inexpensive printed pamphlets.', 'Many believed that the new scientific discoveries justified a more tolerant and objective approach to social and cultural issues.', 'Many were accomplished scientists, who added important pieces to human understanding of the universe.']\",\"question\":\"This question refers to the following information.\\nThe following quote is from Voltaire in response to the 1755 Lisbon earthquake.\\nMy dear sir, nature is very cruel. One would find it hard to imagine how the laws of movement cause such frightful disasters in the best of possible worlds. A hundred thousand ants, our fellows, crushed all at once in our ant-hill, and half of them perishing, no doubt in unspeakable agony, beneath the wreckage from which they cannot be drawn. Families ruined all over Europe, the fortune of a hundred businessmen, your compatriots, swallowed up in the ruins of Lisbon. What a wretched gamble is the game of human life! What will the preachers say, especially if the palace of the Inquisition is still standing? I flatter myself that at least the reverend father inquisitors have been crushed like others. That ought to teach men not to persecute each other, for while a few holy scoundrels burn a few fanatics, the earth swallows up one and all.\\n—Voltaire, in a letter, 1755\\nThe ideas expressed by Voltaire, above, best illustrate which of the following characteristics of Enlightenment intellectuals?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}