{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"44d42a2e-8907-52ba-a575-c5e499715a25","task_key":"all--test--44d42a2e-8907-52ba-a575-c5e499715a25","task_revision_id":"1","upstream_id":"","short_description":"A defendant has been charged with committing the armed robbery of a convenience…","config":"all","split":"test","body":"{\"choices\":\"['admissible, because it is relevant to the crime charged.', 'admissible, because the defendant opened the door by introducing evidence of his good character.', 'inadmissible, because the defendant will be able to prevent the testimony using the spousal testimony privilege.', 'inadmissible, because character evidence may not be introduced in this manner.']\",\"question\":\"A defendant has been charged with committing the armed robbery of a convenience store. During the defendant's case-in-chief, the defendant called several character witnesses to testify as to his good character. As part of the prosecution's rebuttal, the defendant's ex-wife was called to testify that during their marriage the defendant had a violent temper, beat her frequently, and once threatened her with a knife. The ex-wife's testimony should be ruled\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}