{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dcb9d525-e153-5ff3-ab2a-0b2c3f19126f","task_key":"all--test--dcb9d525-e153-5ff3-ab2a-0b2c3f19126f","task_revision_id":"1","upstream_id":"","short_description":"In spite of the fact that Socrates claims to be ignorant of the essence or…","config":"all","split":"test","body":"{\"choices\":\"['justice, like knowledge, requires that we admit that we know nothing and never will.', 'he knows what he is supposed to be looking for--knowledge of the essences of things.', 'knowledge of the essences of things is impossible, because that would require that we know what we are looking for before we know what it is we are looking for.', 'his method of asking questions about essences is itself unjustified because he does not know why he engages in such a practice.']\",\"question\":\"In spite of the fact that Socrates claims to be ignorant of the essence or nature of certain things like justice, he is wise insofar as he recognizes that without such knowledge actions are rationally unjustified.  That is, his wisdom consists in his recognition not only that he is ignorant of such essences but also that:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}