{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"13ee179b-a9b1-5768-921a-fec1ad30ce68","task_key":"default--test--11050","task_revision_id":"1","upstream_id":"11050","short_description":"According to rule consequentialism, the rightness or wrongness of an action…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"the individual's personal beliefs and morals at the time of the action.\\\", 'whether a virtuous person would endorse a rule requiring, permitting, or prohibiting that action.', \\\"that action's relationship to the operative rules of law.\\\", \\\"the action's alignment with universally accepted human rights.\\\", 'the immediate consequences of the action, regardless of any established rules.', 'the logical consistency behind the motive of actions of the same type.', 'whether a religious doctrine approves or disapproves of the action.', 'whether the action aligns with the principles of utilitarianism.', 'the societal norms and customs relevant to the action.', 'whether that action is required, permitted, or prohibited by a rule the consequences of which are best.']\",\"question\":\" According to rule consequentialism, the rightness or wrongness of an action depends on\",\"src\":\"ori_mmlu-moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}