{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4c933b6a-5e16-5a60-8dcb-3da9492ad27c","task_key":"default--test--10999","task_revision_id":"1","upstream_id":"10999","short_description":"In responding to terrorism, says Walzer, it is particularly important that","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['we isolate the terrorists and cut off their resources.', 'we focus on increasing security measures rather than retaliation.', 'we use the same tactics as terrorists to understand their methods.', 'we attack the lower-ranking terrorists first.', 'we take immediate military action against the terrorists.', 'we make an effort to convince the terrorists that they are acting wrongly.', \\\"we publicly condemn the terrorists' actions.\\\", 'we avoid harming terrorists to the same extent that we would avoid harming innocent civilians.', 'we negotiate with the terrorists to reach a peaceful resolution.', 'we do not repeat the wrongs of terrorism in the process.']\",\"question\":\" In responding to terrorism, says Walzer, it is particularly important that\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}