{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d9abd46-298b-578a-864f-1c265f119df9","task_key":"default--test--11003","task_revision_id":"1","upstream_id":"11003","short_description":"The question of whether the killing/letting die distinction is morally relevant…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['killing them by blocking access to clean water.', 'killing them by sending poisoned food.', 'refusing to share agricultural technology with them.', 'withholding medicine that they need to survive.', 'exploiting their resources without fair compensation.', 'denying them the opportunity to trade fairly in global markets.', 'depriving them of knowledge they need to produce food.', 'imposing economic sanctions that lead to food shortages.', 'depriving them of their land for farming.', 'refusing to give them the money they need to purchase food.']\",\"question\":\" The question of whether the killing/letting die distinction is morally relevant can be posed by asking whether we are equally to blame for allowing people in Third World countries to starve as we would be for\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}