{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1f32e430-ee1c-5de3-a776-36d01bdc7d43","task_key":"default--test--11219","task_revision_id":"1","upstream_id":"11219","short_description":"The version of utilitarianism that Brandt defends makes rightness a function of:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the amount of harm caused by an action.', 'whether or not one’s maxim can be universalized.', 'the moral character of the person performing the action.', 'the distribution of wealth in society.', 'the consequences of particular acts.', 'the intentions behind the action.', 'ideal rules.', 'the preferences of the individual.', 'the total amount of happiness in society.', 'the actual and recognized rules of society.']\",\"question\":\"The version of utilitarianism that Brandt defends makes rightness a function of:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}