{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5f17c35e-0031-5bb1-acb9-b3b06893bf34","task_key":"default--test--1253","task_revision_id":"1","upstream_id":"1253","short_description":"A foreign visitor was on trial for kidnapping a small child. The prosecutor…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, the statement does not directly implicate the accused in the crime.', \\\"No, the statement will serve to corroborate the prosecutor's theory of the case.\\\", 'Yes, the statement is irrelevant to the issue of guilt or innocence.', 'Yes, the statement should be excluded because it could bias the jury.', 'Yes, the prejudice of the statement will greatly outweigh its probative value.', 'No, the statement can come in as impeachment if the accused takes the stand, and as a party admission to show the material facts of knowing the child and that the family had money.', 'Yes, the statement is hearsay and therefore inadmissible.', 'No, the statement helps establish the relationship between the accused and the child.', 'No, the statement is admissible to impeach the accused and establish he is lying on cross, if he takes the stand.']\",\"question\":\"A foreign visitor was on trial for kidnapping a small child. The prosecutor stated that the visitor knew the child personally, which is why the child went with him, and that the perpetrator knew the child's parents had money. The prosecutor called a witness to testify that the perpetrator told the witness \\\"I am looking forward to visiting with the child and his parents. They have a wonderfully luxurious house that I will enjoy staying in.'' The defense objected to the proposed testimony. Will the court likely sustain the objection?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}