{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"de0f4577-4647-59fa-95ba-ae804642c9d1","task_key":"default--test--10913","task_revision_id":"1","upstream_id":"10913","short_description":"According to some defenders of meat-eating, the suffering of animals counts less…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['animals can tolerate more suffering than humans can.', 'humans are rational and animals are not.', 'humans have a soul, while animals do not.', 'animals are rational and humans are not.', 'animals lack the capacity for suffering, and humans do not.', 'animals are not sentient beings, unlike humans.', 'animals do not have emotions, while humans do.', 'humans are superior to animals in the food chain.', 'animals are designed by nature to endure suffering.']\",\"question\":\" According to some defenders of meat-eating, the suffering of animals counts less than the suffering of humans (if at all) because\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}