{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"849d6a5a-ea02-52bc-8b30-f9c1b662652f","task_key":"default--test--10894","task_revision_id":"1","upstream_id":"10894","short_description":"Gauthier claims that moral agreements that are equally favorable to all parties…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['promote a sense of community and unity.', 'satisfy our desire for fairness.', 'fulfill the basic needs of all parties.', 'promote the principle of equality.', 'maximize the productivity of society.', 'ensure the preservation of individual rights.', 'increase the overall happiness of society.']\",\"question\":\"Gauthier claims that moral agreements that are equally favorable to all parties are desirable because they:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}