{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"390895df-ce05-5cec-8328-6d5402a84dd9","task_key":"default--test--1826","task_revision_id":"1","upstream_id":"1826","short_description":"A victim and a defendant both worked as longshoremen at a shipyard. After the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"admissible, because it reports the defendant's adoptive admission of the victim's assertion.\\\", 'inadmissible, because it is a character evidence.', 'admissible, because it is considered a prior bad act.', \\\"inadmissible, because it violates the defendant's right to confront the witnesses against him.\\\", 'inadmissible, because it is a prejudicial evidence.', 'inadmissible, because of the principle of collateral estoppel.', 'admissible, because it is a statement made by a party opponent.', 'admissible, because it reports a declaration against interest.', 'inadmissible, because it is hearsay not within any recognized exception.', 'admissible, because it constitutes an excited utterance.']\",\"question\":\"A victim and a defendant both worked as longshoremen at a shipyard. After the victim was shot to death, the defendant was tried for murder and acquitted. Following the acquittal, the victim's estate sued the defendant in a wrongful death action. During the civil trial, the victim's estate called a witness to testify. The witness, who worked with both men at the shipyard, testified that two weeks before the shooting, the victim came to work with a broken nose and said that the defendant had caused it. The attorney for the victim's estate then asked the witness the following question, \\\"Was the defendant present during your conversation with the victim, and if so, did he say anything about the victim's broken nose?\\\" The witness replied, \\\"Yes, the defendant was present, and after the victim told me that the defendant broke his nose, the defendant said, €˜And that's only the beginning. \\\"Upon objection by the defendant's attorney, the witness's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}