{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4984e241-d7fa-52a3-8194-0185185c1ab0","task_key":"all--test--4984e241-d7fa-52a3-8194-0185185c1ab0","task_revision_id":"1","upstream_id":"","short_description":"A toxicologist employed at the state crime laboratory testifies at a murder…","config":"all","split":"test","body":"{\"choices\":\"[\\\"grant the motion, because the toxicologist's opinion could not be substantiated.\\\", 'grant the motion, because the toxicologist was not properly qualified as an expert.', 'deny the motion, because the toxicologist was qualified as an expert in medical matters.', 'deny the motion, because he was qualified to give an opinion on the ultimate issue of the case.']\",\"question\":\"A toxicologist employed at the state crime laboratory testifies at a murder trial about the autopsy he performed on the victim. His testimony is that he detected traces of succinic acid and choline, the components of succinylcholine, in the victim's brain tissues. Over defense objections, he testified further that, in his opinion, the victim's cause of death was succinylcholine poisoning. On cross-examination, the toxicologist admitted that up until his findings, it had been universally accepted in the medical community that succinylcholine would be undetectable in the brain within a few minutes of its injection. He also admitted that his methods of detecting succinylcholine were experimental and that his results were not presently publishable because they were not complete enough. The defense attorney then makes a motion to strike the toxicologist's testimony regarding the cause of death. The court will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}