{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4bc53f0d-3715-5e57-9c36-09170e45e7a7","task_key":"default--test--4692","task_revision_id":"1","upstream_id":"4692","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['Thomas Hobbes because he thought human nature was naturally cruel and selfish.', 'Jean-Jacques Rousseau because he thought society corrupted noble souls.', 'Mary Wollstonecraft because she railed against the subjugation of women emotionally.', 'Denis Diderot because he believed emotions were superior to reason.', 'John Locke because he thought individuals were corrupted by power.', 'Immanuel Kant because he thought that reason was insufficient to explain the mysteries of the universe.', 'Adam Smith because he stated that reason could not be used to understand how another feels.', 'Voltaire himself because he criticized the cruelty of nature and human suffering.', 'Baron Montesquieu because he thought that governments were not capable of being fair.', 'David Hume because he believed that reason was inadequate to make moral judgments.']\",\"question\":\"This question refers to the following information.\\nThe following quote is from Voltaire in response to the 1755 Lisbon earthquake.\\nMy dear sir, nature is very cruel. One would find it hard to imagine how the laws of movement cause such frightful disasters in the best of possible worlds. A hundred thousand ants, our fellows, crushed all at once in our ant-hill, and half of them perishing, no doubt in unspeakable agony, beneath the wreckage from which they cannot be drawn. Families ruined all over Europe, the fortune of a hundred businessmen, your compatriots, swallowed up in the ruins of Lisbon. What a wretched gamble is the game of human life! What will the preachers say, especially if the palace of the Inquisition is still standing? I flatter myself that at least the reverend father inquisitors have been crushed like others. That ought to teach men not to persecute each other, for while a few holy scoundrels burn a few fanatics, the earth swallows up one and all.\\n—Voltaire, in a letter, 1755\\nWhich of the following Enlightenment philosophes questioned reliance on reason and began the Romantic Movement for the rationale given?\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}