{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0e5b0e69-dedf-591d-93a6-6d4b675b6b33","task_key":"all--test--0e5b0e69-dedf-591d-93a6-6d4b675b6b33","task_revision_id":"1","upstream_id":"","short_description":"In a suit between a plaintiff and a defendant arising out of an automobile…","config":"all","split":"test","body":"{\"choices\":\"['permit the jury to consider the prior statement as substantive evidence.', 'permit the defendant to call other witnesses to confirm the fact that the prior statement had been made.', \\\"on request by the defendant, instruct the jury that the bystander's testimony that the defendant had driven through the red light bedisregarded.\\\", \\\"on request by the plaintiff, instruct the jury that the prior statement may be used only toimpeach the bystander's credibility, and not as substantive evidence.\\\"]\",\"question\":\"In a suit between a plaintiff and a defendant arising out of an automobile accident, a bystander wascalled to the stand to testify that the defendant had driven through a red traffic light immediately before the collision. On cross-examination by the defense, the bystander admitted to having made a prior statement to a friend that he was not sure if the traffic signal was yellow or red when the defendant drove through the intersection. The trial judge should\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}