{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c50be7a4-3aa8-5367-b1af-ab8ebaa97d31","task_key":"default--test--1208","task_revision_id":"1","upstream_id":"1208","short_description":"After being passed over for a promotion, an aeronautic engineer became a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['admissible as former testimony.', 'inadmissible as hearsay within the business records exception.', \\\"inadmissible due to the engineer's unavailability.\\\", 'admissible as a statement against interest.', 'admissible as a vicarious admission.', 'inadmissible, because the company did not have the opportunity to cross-examine the engineer on the liability issue for which the statement is now being offered.', 'admissible as a dying declaration.', \\\"inadmissible because the engineer's deposition is biased.\\\", 'inadmissible as hearsay not within any recognized exception.', 'admissible as a statement of personal or family history.']\",\"question\":\"After being passed over for a promotion, an aeronautic engineer became a whistleblower. He contacted a government agency to state that the company for which he worked was submitting false safety reports on newly built aircraft. When the company learned that the engineer was leaking this information to the government agency, he was fired from his job. Afterward, the engineer sued the company for wrongful termination of employment. During the discovery stage of litigation, the engineer was deposed by the company's attorney. In his deposition, the engineer stated that the company submitted false safety reports to the government agency to cover up structural defects in its aircraft. A pilot was injured when one of the company'sairplanes he was piloting crashed. The pilot asserteda strict products liability tort against the company. At thal, the pilot seeks to introduce into evidence portions of the engineer's deposition from his wrongful termination lawsuit against the company. Assume that the engineer is unavailable to testify at trial. Upon objection by the company, the trial court judge should rule the engineer's deposition testimony\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}