{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"faac7215-dead-524e-a068-0ab6aee9f045","task_key":"default--test--1102","task_revision_id":"1","upstream_id":"1102","short_description":"A husband and his wife are involved in a contested divorce and child custody…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admitted, because the spousal privilege is inapplicable, since the parties are living in separation.', \\\"excluded, because the wife cannot testify about confidential information she shared with the husband's attorney.\\\", \\\"admitted, because the attorney-client privilege was waived when the wife shared information in the husband's absence.\\\", \\\"admitted, because the wife's testimony is considered direct evidence, not privileged communication.\\\", 'excluded, because the wife is incompetent to testify to marital communications because she is not the holder of the privilege.', \\\"excluded, because the wife's testimony can be considered hearsay.\\\", 'admitted, because the information is crucial to the case and overrides any privilege.', 'admitted, because the marital communication privilege is inapplicable in a lawsuit involving the parties.', 'excluded, because confidential statements made during a joint consultation are privileged communications between clients and their attorneys.']\",\"question\":\"A husband and his wife are involved in a contested divorce and child custody battle. Ignorant of the adversarial system, they both visited a family law attorney together for an initial consultation. The attorney advised them that he could not represent them both. The wife found another attorney. During the trial, the wife's attorney calls the wife to testify. She states that during the initial consultation she and her husband had with his attorney, she privately told the attorney, when her husband was taking a bathroom break, that her husband had a bad drinking problem, which was one of her major reasons for seeking a divorce and custody of the children. Upon objection by the husband's attorney, the wife's testimony should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}