{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f0133dbf-3dc0-5cc2-b155-e807568f37d7","task_key":"default--test--934","task_revision_id":"1","upstream_id":"934","short_description":"A 35-year-old man with some experience as a truck driver owned a lumber truck.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"not recover, because the man had no control over the hospital's actions.\\\", 'recover, because the man failed to properly instruct the helper on emergency procedures.', \\\"not recover, because the man's actions were intended to save the helper's life.\\\", \\\"not recover, because the nurse's negligence in giving the wrong medication is unrelated to the man's actions.\\\", \\\"recover, because the man was responsible for the helper's safety while in the truck.\\\", \\\"recover, because the man was negligent in his maintenance of the truck's brakes.\\\", 'recover, because the man was negligent in pushing the helper out of the truck.', 'not recover, because it is not foreseeable that a patient would be injured in such a manner.', 'recover, because it is foreseeable that a hospital can be negligent in its care of patients.']\",\"question\":\"A 35-year-old man with some experience as a truck driver owned a lumber truck. One day, the man set out driving his truck, heavily loaded with lumber, down a mountain road. Sitting next to the man in the passenger seat was a 19-year-old helper. During the course of the trip, when the truck was going down a long hill, the brakes failed. The man shouted to the helper to jump, but the teenager refused to do so and shouted back to the man that he should try to steer the truck down the hill. The man then opened the door on the passenger's side of the truck and negligently pushed the helper out. The helper, who suffered a broken leg, was rushed to the hospital where he was treated for his injury. As the helper was recuperating, a nurse inadvertently mixed up his chart with that of the female patient in the next room. The nurse gave the helper a fertility pill that made him sterile. If the helper brings suit against the man to recover damages for his sterility, the man will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}