{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"53fec05e-c8a6-5d09-b9c8-eef4a30b05ba","task_key":"default--test--954","task_revision_id":"1","upstream_id":"954","short_description":"A plaintiff sued the insurer of her home after the insurer denied coverage for…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"None of the testimony is admissible, because it is collateral, having no bearing on the engineer's qualifications as an expert.\\\", \\\"Only the portion concerning the engineer's reputation is admissible, because where both opinion and reputation evidence are available, only the latter is admissible under a rule of preference.\\\", \\\"Both portions of the testimony are admissible, but only as they relate to the engineer's truthfulness, not his expertise.\\\", \\\"None of the testimony is admissible, because character evidence is not relevant to the engineer's expertise.\\\", \\\"All of the testimony is admissible to support the engineer's credibility.\\\", \\\"Only the portion concerning the engineer's reputation is admissible, as it is directly relevant to his character for truthfulness.\\\", \\\"Only the portion concerning the witness's opinion of the engineer's character is admissible, as it directly relates to his credibility.\\\", \\\"Only the portion concerning the witness's opinion of the engineer's character, because the witness's reporting of the neighbors' comments is hearsay.\\\", 'None of the testimony is admissible, because the witness is biased due to their acquaintance with the engineer.', \\\"All of the testimony is admissible, but the judge must instruct the jury to consider the engineer's past conviction.\\\"]\",\"question\":\"A plaintiff sued the insurer of her home after the insurer denied coverage for water damage to the home allegedly caused by a frozen plastic pipe that burst. At trial, the insurer called as an expert witness an engineer, who testified that the pipe had burst because of age rather than freezing. On cross-examination, the engineer admitted that five years earlier, he had been convicted of tax fraud, even though he had asserted that it was his accountant's error. In response, the insurer calls a witness, who is well acquainted with the engineer and his reputation, to testify that (1) in the witness's opinion, the engineer is a truthful person, and (2) the engineer's neighbors all describe him as a truthful person. How much, if any, of the witness's testimony is admissible?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}