{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c003ef2f-540a-580f-9459-bb7c61efee31","task_key":"default--test--11028","task_revision_id":"1","upstream_id":"11028","short_description":"In spite of the fact that Socrates claims to be ignorant of the essence or…","config":"default","split":"test","body":"{\"category\":\"philosophy\",\"options\":\"['justice, like knowledge, requires that we admit that we know nothing and never will.', 'he is wise because he knows that he can learn from others.', 'his wisdom is based on his ability to question everything and everyone.', 'his method of asking questions about essences is itself unjustified because he does not know why he engages in such a practice.', \\\"his wisdom is in knowing that he doesn't need to know the essence of things to be wise.\\\", 'the essence of things is not important for him to be wise.', 'he knows what he is supposed to be looking for--knowledge of the essences of things.', 'his wisdom lies in knowing that he knows more than he thinks he does.']\",\"question\":\"In spite of the fact that Socrates claims to be ignorant of the essence or nature of certain things like justice, he is wise insofar as he recognizes that without such knowledge actions are rationally unjustified.  That is, his wisdom consists in his recognition not only that he is ignorant of such essences but also that:\",\"src\":\"ori_mmlu-philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}