{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"891dbac3-c407-5b74-8a12-783cdae05edb","task_key":"default--test--11045","task_revision_id":"1","upstream_id":"11045","short_description":"Which of the following potentially morally relevant differences between Fred's…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['Fred\\\\'s behavior involves the suffering of puppies, whereas the behavior of meat-eaters \\\"merely\\\" involves the suffering of chickens, pigs, cows, calves, sheep, and the like.', 'The suffering of the puppies is necessary for gustatory pleasure, whereas the suffering of animals on factory farms is simply a by-product of the conditions dictated by economic considerations.', \\\"Fred could prevent the suffering of the puppies, whereas meat-eaters can't prevent the suffering of the animals they consume.\\\", 'none of the above']\",\"question\":\" Which of the following potentially morally relevant differences between Fred's behavior and the behavior of the consumers of factory-raised meat does Norcross not discuss?\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}