{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e15d1ba6-75e2-5ce6-8bd6-5f6bcf68baad","task_key":"all--test--e15d1ba6-75e2-5ce6-8bd6-5f6bcf68baad","task_revision_id":"1","upstream_id":"","short_description":"A witness sees a murder in the park and is called to testify in the criminal…","config":"all","split":"test","body":"{\"choices\":\"['No, because the question has no probative value regarding the credibility of the witness or the guilt of the defendant.', 'No, because a witness can never be asked about personal prior bad acts.', \\\"Yes, because the statement indicates that the witness' attention was always focused on feeding the pigeons.\\\", 'Yes, because feeding pigeons in the park and persisting in it after being ticketed may show an antisocial trait of the witness.']\",\"question\":\"A witness sees a murder in the park and is called to testify in the criminal case. Can the defense attorney ask her on cross examination about her habit of feeding the pigeons? The defense knows that she was cited several times for illegally feeding the pigeons. Defense counsel tries to ask, \\\"You have a longtime habit of feeding the pigeons, don't you? You have in fact been ticketed several times for feeding pigeons, haven't you, and yet you persisted in doing it, isn't that so?\\\" Will the trial court likely allow the witness to be questioned on these matters?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}