{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8cfd9a65-8028-5589-a1ab-b9368bd3f48c","task_key":"default--test--11237","task_revision_id":"1","upstream_id":"11237","short_description":"According to Sinnott-Armstrong, the fact your government morally ought to do…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['does not prove that government officials ought to promote it.', 'does not prove that you ought to do it.', 'proves that you ought not to do it.', 'proves that you ought to do it, too.']\",\"question\":\" According to Sinnott-Armstrong, the fact your government morally ought to do something\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}