{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0b35c2be-4707-50ca-85e9-646f942e83aa","task_key":"default--test--11155","task_revision_id":"1","upstream_id":"11155","short_description":"Cicero claims that there could be “no more pernicious doctrine” than the idea…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"[\\\"it is sometimes necessary to act against one's principles.\\\", 'it is sometimes permissible to lie.', 'moral rightness is relative to one’s culture.', 'morality is subjective and can be changed based on personal preferences.', 'a thing may be expedient without being morally right.', 'lying is always morally wrong.', \\\"one's actions do not have consequences.\\\", 'the gods do not care what we do.', 'there is no absolute truth.', 'the gods are indifferent to human affairs.']\",\"question\":\"Cicero claims that there could be “no more pernicious doctrine” than the idea that:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}