{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7cd23124-27f9-5796-8d82-3910ace2d1ec","task_key":"default--test--1694","task_revision_id":"1","upstream_id":"1694","short_description":"A year ago, a very wealthy man died in a car accident. A woman, whose mother…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"sustain the objection because the butler's testimony is considered privileged communication and is therefore inadmissible.\\\", \\\"overrule the objection because the butler's testimony would be relevant to the ultimate issue in the case.\\\", \\\"sustain the objection because the butler's testimony would be considered gossip and not admissible in court.\\\", \\\"sustain the objection bc butler's testimony would be violative of hearsay rule.\\\", \\\"overrule the objection because the butler's testimony is considered a statement against interest and is therefore admissible.\\\", \\\"overrule the objection because the butler's testimony qualifies under the exception to the hearsay rule for statements made for purposes of medical diagnosis or treatment.\\\", \\\"sustain the objection because the butler's testimony would be in violation of attorney-client privilege.\\\", \\\"overrule the objection because the butler's testimony would be admissible as an exception to the hearsay rule under the dying declaration doctrine.\\\", \\\"overrule objection because the butler's testimony would qualify under the pedigree exception to the hearsay rule.\\\", \\\"sustain, bc butler's testimony would be violative of Dead man's statute.\\\"]\",\"question\":\"A year ago, a very wealthy man died in a car accident. A woman, whose mother worked as a secretary to the wealthy man for years, filed a claim against the estate, alleging that she was the wealthy man's daughter. At trial, the woman's attorney called the wealthy man's butler to testify. The butler testified that it was Common knowledge in the household that the wealthy man and the woman's mother were having an affair. The butler also testified that the wealthy man had often confided in him and told him shortly after the woman's birth that the woman was the wealthy man's daughter. Upon objection by the attorney for the executor of the wealthy man's estate, the court will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}