{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4c171969-0f9b-54df-bfe1-93957c850687","task_key":"default--test--11184","task_revision_id":"1","upstream_id":"11184","short_description":"Mill claims that the distinction between justice and other moral obligations…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['strong and weak duties.', 'personal and public duties.', 'conscious and unconscious obligations.', 'absolute and relative obligations.', 'positive and negative duties.', 'moral and ethical obligations.', 'direct and indirect duties.', 'legal and illegal obligations.', 'perfect and imperfect obligations.', 'urgent and non-urgent obligations.']\",\"question\":\"Mill claims that the distinction between justice and other moral obligations corresponds perfectly to the distinction between:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}