{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7739d049-37dd-55dd-a835-00a230147dcf","task_key":"default--test--968","task_revision_id":"1","upstream_id":"968","short_description":"A football player sued a football team for breach of contract. He was a star…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['inadmissible, because the team physician is not a radiologist capable of interpreting MRI results.', \\\"admissible, if the football player confirms that the physician's diagnosis was shared with him.\\\", 'admissible, provided that the other doctor testifies as to the validity of the MRI test results.', 'admissible, provided that the team physician has been practicing for more than 10 years.', \\\"inadmissible, because the team physician's opinion was not based in part upon his own firsthand observation of the player's physical condition.\\\", 'inadmissible, because neither the x-rays nor the MRI test results had been admitted into evidence at the time he formulated his opinion.', 'admissible, only if the x-rays and MRI test results are presented during the trial.', 'admissible, provided the team physician based his opinion on data reasonably relied upon by other orthopedic physicians in the field.', 'inadmissible, as the team physician has a possible bias towards the football team.', 'inadmissible, because the team physician should have conducted the MRI test himself.']\",\"question\":\"A football player sued a football team for breach of contract. He was a star player for the team before suffering a serious hip injury. Although the player attempted to continue playing for the team, he failed a physical examination and was cut from the team roster. After failing his physical, the team discontinued making contract payments to the player as per the terms of his employment contract. In his lawsuit, the player claimed that he was physically fit to continue playing but that he was cut from the team because they wanted to stop making contract payments. At trial, the team called its physician, an orthopedist, to testify that the player's hip injury was, in fact, career-threatening and, in his opinion, sufficiently serious to warrant failing his physical. The team physician stated during his testimony that his diagnosis was based, in part, upon an examination of the player's x-rays and review of an MRI test conducted by another doctor. Neither the x-rays nor the MRI test results have been admitted into evidence. In all likelihood, the team physician's testimony should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}