{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"849554e0-fd06-5d8f-b8af-8a344de595aa","task_key":"all--test--849554e0-fd06-5d8f-b8af-8a344de595aa","task_revision_id":"1","upstream_id":"","short_description":"A defendant, on trial for robbery, took the stand and testified in his own…","config":"all","split":"test","body":"{\"choices\":\"[\\\"admissible, as bearing on the defendant's credibility.\\\", \\\"admissible, as bearing on the defendant's guilt.\\\", 'inadmissible, because the testimony is extrinsic evidence.', 'inadmissible, because the court order is thebest evidence.']\",\"question\":\"A defendant, on trial for robbery, took the stand and testified in his own behalf. On cross-examination, the prosecuting attorney asked the defendant whether he had committed false pretenses by making long-distance telephone calls by the unauthorized use of a credit card number. The defendant replied, \\\"That's a lie, I never did. \\\" In rebuttal, the prosecuting attorney called the court officer to testif' that the defendant admitted committing false pretenses. The court officer further testified, however, that the defendant was granted immunity from prosecution in exchange for his testimony against others. The court officer's testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}