{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"215319f5-a36a-5b98-839f-a6254d117356","task_key":"all--test--215319f5-a36a-5b98-839f-a6254d117356","task_revision_id":"1","upstream_id":"","short_description":"According to Socrates, just as there is a difference between what an ironic…","config":"all","split":"test","body":"{\"choices\":\"['what is right or wrong, true or false varies from one culture to another.', 'appearances are the only real way we have for knowing reality.', 'the distinction of appearance and reality is the basis for the dialectical discovery of truth.', 'there are objective principles for thought and action that are required for the good life.']\",\"question\":\"According to Socrates, just as there is a difference between what an ironic statement says and its true meaning, so also appearances differ from reality.  Even though societies or individuals appear to differ about what is required for the good life, that in no way contradicts the fact that:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}