{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"37e61dc2-415e-5159-bae3-bd555deb853b","task_key":"default--test--1021","task_revision_id":"1","upstream_id":"1021","short_description":"A toxicologist employed at the state crime laboratory testifies at a murder…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['deny the motion, because he was qualified to give an opinion on the ultimate issue of the case.', \\\"deny the motion, because the toxicologist's findings were based on his professional expertise.\\\", 'deny the motion, because the toxicologist was qualified as an expert in medical matters.', 'deny the motion, because the court values the opinion of the state-employed expert.', \\\"grant the motion, because the toxicologist's opinion could not be substantiated.\\\", \\\"deny the motion, because the toxicologist's opinion is the only evidence of the cause of death.\\\", 'grant the motion, because the toxicologist was not properly qualified as an expert.', 'grant the motion, because the toxicologist admitted his findings contradicted accepted medical knowledge.', \\\"grant the motion, because the toxicologist's results were not complete enough to be published.\\\"]\",\"question\":\"A toxicologist employed at the state crime laboratory testifies at a murder trial about the autopsy he performed on the victim. His testimony is that he detected traces of succinic acid and choline, the components of succinylcholine, in the victim's brain tissues. Over defense objections, he testified further that, in his opinion, the victim's cause of death was succinylcholine poisoning. On cross-examination, the toxicologist admitted that up until his findings, it had been universally accepted in the medical community that succinylcholine would be undetectable in the brain within a few minutes of its injection. He also admitted that his methods of detecting succinylcholine were experimental and that his results were not presently publishable because they were not complete enough. The defense attorney then makes a motion to strike the toxicologist's testimony regarding the cause of death. The court will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}