{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1e5be076-ff4f-5673-ba36-73fa5dd30f01","task_key":"default--test--10790","task_revision_id":"1","upstream_id":"10790","short_description":"The principle Arthur calls \"the greater moral evil rule\"says that","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['morality itself is a great evil.', 'people should always strive to commit the lesser evil, even if it results in personal gain.', 'the person who commits the greater evil is the one who is morally responsible for the situation.', 'people are entitled to keep their earnings only if there is no way for them to prevent a greater evil by giving them away.', 'people are morally obligated to share their wealth if it can prevent a greater evil.', 'being rich is a great moral evil, and as such we should all strive to live on a moderate income.', \\\"the greater moral evil rule is a concept which states that it's always wrong to do evil, regardless of the circumstances.\\\", 'two moral evils do not make a moral good.', 'people are allowed to do a smaller evil to prevent a greater one.', 'the concept of evil is subjective and varies from person to person.']\",\"question\":\" The principle Arthur calls \\\"the greater moral evil rule\\\"says that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}