{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"71f554fb-ca56-5394-880f-28b325104ef1","task_key":"all--test--71f554fb-ca56-5394-880f-28b325104ef1","task_revision_id":"1","upstream_id":"","short_description":"According to Socrates, the task of the wise and virtuous person is not simply to…","config":"all","split":"test","body":"{\"choices\":\"['by knowing enough examples of justice or virtue, we will live a worthwhile life even if we do not know what makes them examples of justice or virtue.', 'knowledge of individual examples alone would not prepare someone for situations of justice or virtue to which the examples do not immediately apply.', 'what makes an action just or virtuous can be known only by asking people for their opinions and respecting each answer as equally valuable.', 'justice and virtue are universal goals of all human beings, even if people do not always agree on how to achieve those ends.']\",\"question\":\" According to Socrates, the task of the wise and virtuous person is not simply to learn various examples of just or virtuous actions but to learn the essence of justice or virtue, because:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}