{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"99c7e955-145a-5080-bff9-aefacbdc59f3","task_key":"all--test--99c7e955-145a-5080-bff9-aefacbdc59f3","task_revision_id":"1","upstream_id":"","short_description":"Plato indicates that the knowledge of pure reason is preferable to conceptual…","config":"all","split":"test","body":"{\"choices\":\"['how we come to learn what to call a thing in virtue of our own experiences.', 'the logos or rationale of the thing, that is, why it is the way it is.', 'why we differ among ourselves about what we claim to know.', \\\"the difference between knowledge and opinion as outlined in Plato's divided line image.\\\"]\",\"question\":\" Plato indicates that the knowledge of pure reason is preferable to conceptual understanding, because knowing that something is a certain kind of thing is not as good as knowing:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}