{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a48fad76-4625-5be9-bb96-8ce4d49886a8","task_key":"default--test--2337","task_revision_id":"1","upstream_id":"2337","short_description":"For deception in an experiment to be permissible, it is essential that the","config":"default","split":"test","body":"{\"category\":\"psychology\",\"options\":\"['the participants have given their informed consent to be deceived', 'deception does not involve withholding information about the purpose of the research', 'investigator include a careful explanation of the deception within 3 years following the experiment', 'merits or value of the scientific investigation warrants the type of deception used', 'the deception involves providing false information about the purpose of the research', 'the deception is revealed to the participants at the end of the study, but only if they ask']\",\"question\":\"For deception in an experiment to be permissible, it is essential that the\",\"src\":\"ori_mmlu-professional_psychology\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}