{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cc29675e-6f9f-5183-b238-a786b6ee9d48","task_key":"default--test--10895","task_revision_id":"1","upstream_id":"10895","short_description":"Anscombe criticizes Sidgwick on the grounds that:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['he supports the principle of consequentialism.', 'he endorses the divine law conception of ethics.', 'he claims that there is a “natural balance” to the universe.', 'he does not believe in the principle of double effect.', 'all of the above.', 'he neglects the role of emotions in ethical decision-making.', 'he fails to account for the role of virtue in ethics.', 'he does not distinguish between the foreseen and intended consequences of one’s action.', 'he argues that morality is subjective.', 'he overlooks the importance of individual intention in moral actions.']\",\"question\":\"Anscombe criticizes Sidgwick on the grounds that:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}