{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"499bc486-be1f-5f0c-b038-c8d536bac502","task_key":"default--test--10850","task_revision_id":"1","upstream_id":"10850","short_description":"One objection to Singer’s theory that he considers is that it:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['inappropriately makes proximity morally important.', 'fails to consider the cultural differences in moral norms.', 'overlooks the role of government and institutions in addressing suffering.', 'does not do enough to address suffering in other countries besides our own.', 'does not account for the fact that we have stricter obligations to our loved ones than to strangers.', 'does not address the issue of moral obligations to non-human animals.', 'ignores the importance of self-interest in human behavior.', 'assumes that all individuals have the same capacity to alleviate suffering.', 'does not provide a clear definition of what constitutes suffering.', 'requires too drastic a revision to our moral scheme.']\",\"question\":\"One objection to Singer’s theory that he considers is that it:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}