{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"70d4fd68-6863-5cec-ae63-0fcbb6c40bbb","task_key":"default--test--1366","task_revision_id":"1","upstream_id":"1366","short_description":"The Farstate Department of Wealth Equality is a state agency dedicated to taking…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The action is not deemed done by a state actor because wealth redistribution is not an essential government function.', \\\"The action is unconstitutional because it violates the employees' right to equal protection.\\\", 'The rule is not discriminatory because it applies to all employees, regardless of their wealth.', \\\"The action is constitutional because the rule is rationally related to the department's purpose.\\\", 'Wealth is a quasi-suspect class, so the rule must promulgate an important governmental purpose. Eliminating poverty is important, so the rule meets this level of scrutiny.', 'The employees would likely prevail because the rule is not narrowly tailored to achieve a compelling governmental interest.', 'The action is arbitrary and capricious, so it is unconstitutional under the strict scrutiny standard.', 'The rule is valid because the department has a legitimate interest in ensuring its employees understand the plight of the poor.', 'The employees would not likely prevail because they were not deprived of a protected liberty interest.', 'The right to public employment is a protected property interest and the employees would likely prevail because of lack of procedural due process.']\",\"question\":\"The Farstate Department of Wealth Equality is a state agency dedicated to taking from the rich to give to the poor. The head of the department has promulgated the following hiring rule: \\\"Because the rich are the source of all problems of society and cannot understand the plight of the poor, anyone who has a net worth over $10,000.00 is ineligible for employment with this department.\\\" Immediately, 25 employees in the department are fired due to their net worth above the designated level. They sue the state. Which of the following is the correct analysis and conclusion?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}