{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1928413a-a736-59b2-b993-59c4aa162bc3","task_key":"default--test--11201","task_revision_id":"1","upstream_id":"11201","short_description":"With respect to the issue of world hunger and poverty, Arthur points out that…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the principles of sacrifice and reward.', 'the principles of the right and the good.', 'the principles of contribution and entitlement.', 'the principles of equality and harm.', 'the principles of rights and desert.', 'the principles of need and capacity.', 'the principles of duty and benefit.', 'the principles of ability to help and desires.', 'the principles of obligation and privilege.', 'the principles of justice and fairness.']\",\"question\":\" With respect to the issue of world hunger and poverty, Arthur points out that Singer ignores\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}