{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a1064d72-e437-5f54-8bd4-f1620f315fec","task_key":"default--test--11236","task_revision_id":"1","upstream_id":"11236","short_description":"In Singer's understanding, the principle of the equality of human beings is","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['a description of an alleged actual equality among humans.', 'a description of an alleged equality among all living beings.', 'a prescription of how we should treat nonhuman animals.', 'a description of an alleged inequality among all living beings.', 'a prescription of how we should treat humans.', 'a description of an alleged actual inequality among humans.', 'a description of an alleged actual superiority of humans over nonhuman animals.', 'a prescription of how we should treat both human and nonhuman animals equally.', 'a prescription of how we should treat nonhuman animals differently.', 'a prescription of how we should treat the environment.']\",\"question\":\" In Singer's understanding, the principle of the equality of human beings is\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}