{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fc749bb1-16d0-5969-bcb0-12bc999d20bf","task_key":"default--test--10791","task_revision_id":"1","upstream_id":"10791","short_description":"Ross claims that the essential defect of utilitarianism is that it ignores","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the significance of moral intentions.', 'epistemological responsibility.', 'the role of virtue in morality.', 'the value of pleasure.', 'the importance of individual rights.', 'the potential consequences of actions.', 'the concept of justice.', 'moral obligations.', 'the personal character of duty.', 'autonomy.']\",\"question\":\" Ross claims that the essential defect of utilitarianism is that it ignores\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}