{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e725aca2-3d77-5c02-b96f-c5abf0f43eea","task_key":"all--test--e725aca2-3d77-5c02-b96f-c5abf0f43eea","task_revision_id":"1","upstream_id":"","short_description":"During his trial, Socrates argues that he would never intentionally corrupt…","config":"all","split":"test","body":"{\"choices\":\"['it is his duty not to do so.', 'this would make them vicious, and they would then harm him.', 'he would be likely to get caught if he did so.', 'he would not know how.']\",\"question\":\"During his trial, Socrates argues that he would never intentionally corrupt others because:\",\"subject\":\"philosophy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}