{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6461ae25-9369-57fd-b894-205f61e910c7","task_key":"default--test--4944","task_revision_id":"1","upstream_id":"4944","short_description":"This question refers to the following information.","config":"default","split":"test","body":"{\"category\":\"history\",\"options\":\"['the rise of the Luddite movement in Britain in the eighteenth century', 'the decline of the British economy in the eighteenth century', 'the protest against child labor in the eighteenth century Britain', 'the rise of trade unions in Britain in the eighteenth century', 'the spread of revolutionary ideas among the British working class in the eighteenth century', 'an increase in the use of child labor in eighteenth-century Britain', 'the replacing of traditional laborers by machines in the industrialization of Britain in the eighteenth century']\",\"question\":\"This question refers to the following information.\\nRead the the following petition to answer questions.\\nThe Scribbling-Machines have thrown thousands of your petitioners out of employ, whereby they are brought into great distress, and are not able to procure a maintenance for their families, and deprived them of the opportunity of bringing up their children to labour. … The number of Scribbling-Machines extending about seventeen miles south-west of Leeds exceed all belief, being no less than one hundred and seventy! And as each machine will do as much work in twelve hours, as ten men can in that time do by hand, … [And, as the machines do] as much work in one day as would otherwise employ twenty men, … [a] full four thousand men are left to shift for a living how they can, and must of course fall to the Parish, if not timely relieved. … How are those men, thus thrown out of employ to provide for their families; and what are they to put their children apprentice to, that the rising generation may have something to keep them at work, in order that they may not be like vagabonds strolling about in idleness? … Many more evils we could enumerate, but we would hope, that the sensible part of mankind, who are not biased by interest, must see the dreadful tendency of their continuance; a depopulation must be the consequence; trade being then lost, the landed interest will have no other satisfaction but that of being last devoured.\\nLeeds Woolen Workers Petition, 1786\\nThis document can be used as evidence for\",\"src\":\"ori_mmlu-high_school_european_history\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}