{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f1614658-2888-5295-b1ca-38e200d0a66c","task_key":"all--test--f1614658-2888-5295-b1ca-38e200d0a66c","task_revision_id":"1","upstream_id":"","short_description":"Defendant was on trial for robbery. Defendant's direct examination continued…","config":"all","split":"test","body":"{\"choices\":\"['proper, because a judge has broad discretion to instruct witnesses in such a manner.', 'proper, because it would have the same effect as permitting cross-examination to continue after direct was concluded.', \\\"improper, because it violates the defendant's Sixth Amendment right to counsel.\\\", 'improper, because it violates the attorney-client privilege.']\",\"question\":\"Defendant was on trial for robbery. Defendant's direct examination continued until late in the day when it finally concluded. At this time, the trial court judge adjourned the proceedings for the evening. The judge then instructed Defendant not to speak with anyone during the night and scheduled cross- examination to begin in the morning. The judge's instruction to Defendant was\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}