{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b5aff1d5-90b8-57f9-81de-1b080048c3b6","task_key":"default--test--1173","task_revision_id":"1","upstream_id":"1173","short_description":"A commissioner of a sports league was charged with embezzling funds by…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible, to show the precedent set by the predecessor.', 'inadmissible, because it is second-hand information.', 'inadmissible as hearsay not within any recognized exception.', \\\"inadmissible, because the league's charter is the best evidence.\\\", \\\"admissible, as it can prove the commissioner's innocence.\\\", \\\"inadmissible, because the predecessor's statement cannot be verified.\\\", 'admissible, to show that the commissioner lacked criminal intent.', 'inadmissible, as it is not relevant to the case.', 'admissible as evidence of the routine practice of an organization.', \\\"admissible, as it gives insight into the league's charter interpretation.\\\"]\",\"question\":\"A commissioner of a sports league was charged with embezzling funds by increasing the amount of his allocated travel expenses without the approval of a majority of the team owners. In accordance with the bylaws of the league's charter, the commissioner was required to obtain the approval of at least half the owners before raising his expense allocation. At trial, the commissioner seeks to testify that his predecessor told him that he had authority under the league's charter to raise his expense allocation. The commissioner's testimony is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}