{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0322d3c4-3cd1-5d25-ac14-4f46578d72e2","task_key":"default--test--10983","task_revision_id":"1","upstream_id":"10983","short_description":"Anscombe criticizes as absurd Kant’s idea of:","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['the postulate of immortality.', 'the autonomous will.', 'the categorical imperative.', 'legislating for oneself.', 'the unity of consciousness.', 'the transcendental unity of apperception.', 'the phenomenal self.', 'the thing in itself.', 'the noumenal self.', 'the kingdom of ends.']\",\"question\":\"Anscombe criticizes as absurd Kant’s idea of:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}