{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2e993f8e-d664-5bfc-9de4-71566f3ddd79","task_key":"default--test--11270","task_revision_id":"1","upstream_id":"11270","short_description":"Huemer worries that the employment of a \"war\" metaphor when describing…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['does not really describe the situation well at all, since it is more like a game than a war.', 'diminishes the seriousness of actual wars.', 'masks a bad argument by analogy.', 'is perfectly apt.', 'makes it hard to think and discuss the rights of drug users.', 'indicates that the issue can only be solved through violent measures.', 'implies that the government is the enemy.', 'suggests that drug users are combatants.']\",\"question\":\" Huemer worries that the employment of a \\\"war\\\" metaphor when describing government policies about drug use\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}