{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c4a693b1-bfe0-5d80-beea-3d03abd8a952","task_key":"all--test--c4a693b1-bfe0-5d80-beea-3d03abd8a952","task_revision_id":"1","upstream_id":"","short_description":"At trial in a criminal prosecution for theft, the defendant calls a witness to…","config":"all","split":"test","body":"{\"choices\":\"['No, because it is irrelevant to the present charge.', 'No, because the defendant may not prove his good character by specific instances of good conduct.', 'Yes, because a criminal defendant may prove his good character as a basis for inferring conduct.', 'Yes, because, by accusing the defendant of being a thief, the prosecution has put his character in issue.']\",\"question\":\"At trial in a criminal prosecution for theft, the defendant calls a witness to testify that he formerly knew the defendant as an army supply sergeant and that the defendant had turned down many opportunities for black marketeering. Is the witness's testimony admissible?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}