{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e61c5c3e-3fb2-5f3f-969d-d8319782c6d7","task_key":"default--test--5050","task_revision_id":"1","upstream_id":"5050","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['believed that working people should unite for a common cause', 'believed that government should ensure satisfaction of its citizens', 'supported government intervention in economic matters', 'advocated for higher taxes to support social causes', 'believed that the government should do more than merely protect its people', 'believed in a society where the strong help the weak', 'opposed the use of tax money to provide aid to the poor', 'advocated for a welfare state']\",\"question\":\"This question refers to the following information.\\nThe assumption by a government of the office of Reliever-general to the poor is necessarily forbidden by the principle that a government cannot rightly do anything more than protect. In demanding from a citizen contributions for the mitigation of distress … the state is … reversing its function. … To enforce the fundamental law—to take care that every man has freedom to do all that he wills, provided he infringes not the equal freedom of any other man. … [But that] is quite a separate thing from insuring him satisfaction. …\\nThe poverty of the incapable, the distresses that come upon the imprudent, the starvation of the idle, and those shoulderings aside of the weak by the strong … are the decrees of a large, farseeing benevolence. … When regarded not separately, but in connection with the interests of universal humanity, these harsh fatalities are seen to be full of the highest beneficence.\\nHerbert Spencer, Social Statistics: Survival of the Fittest Applied to Humankind, 1851\\nFrom the passage, one may infer that Spencer\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}