{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d9dd0961-8d18-5824-898d-9f187498680c","task_key":"default--test--1104","task_revision_id":"1","upstream_id":"1104","short_description":"A gardener sued a homeowner for breach of contract. The gardener claims that the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"inadmissible, as it is hearsay and not an exception under the Dead Man's Statute.\\\", 'inadmissible, because the gardener did not have a translator present during the agreement.', \\\"admissible, as it is relevant to the case and not excluded by the Dead Man's Statute.\\\", 'admissible as non-hearsay to corroborate evidence of a verbal act.', 'inadmissible, because the friend was not an official translator.', 'admissible, because it is a statement of a deceased person.', 'admissible as an admission by a party-opponent.', \\\"inadmissible, unless it is established that the gardener accepted the friend as the homeowner's agent in communicating the terms of the contract.\\\", 'admissible, as it is a statement made during the course of employment.']\",\"question\":\"A gardener sued a homeowner for breach of contract. The gardener claims that the homeowner hired him to perform landscape work at his home. After performing the gardening services, the homeowner paid him $100. The gardener, who speaks only English, contends that the homeowner, who speaks only Japanese, originally agreed to pay him $200 for the work. When the parties entered into their agreement, the gardener was accompanied by his friend, who is now deceased, who spoke both English and Japanese. This jurisdiction has a typical Dead Man's Statute in effect. At trial, the gardener now seeks to testify to the terms of his agreement with the homeowner as communicated to him by his friend. Upon objection by the homeowner's attorney, the gardener's proposed testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}