{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f26786fa-abd2-5d1c-81ad-8c74fa82470c","task_key":"default--test--1966","task_revision_id":"1","upstream_id":"1966","short_description":"A surgeon operated on a patient in an operating room in the Marina General…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"prevail, because the nurse's liability is established under res ipsa loquitur\\\", 'not prevail, because the intern was the one who agreed to the settlement.', \\\"not prevail, because the intern's proper remedy is indemnification, not contribution.\\\", 'not prevail, because one who settles without judgment cannot recover contribution.', \\\"not prevail, because the nurse was just following the intern's orders.\\\", 'not prevail, because the intern already accepted a settlement.', 'prevail, because one who settles without judgment can recover contribution.', \\\"prevail, because both were equally responsible for the patient's pain.\\\", 'prevail, because the nurse was also involved in the faulty bandaging.', 'prevail, because the nurse should have known better as a staff member of the hospital.']\",\"question\":\"A surgeon operated on a patient in an operating room in the Marina General Hospital. The surgeon was assisted by an intern, who was assigned to the operation by the hospital, and a nurse, who was on the staff of the hospital. During the patient's convalescence, the patient complained of pain not explicable as an ordinary post-operative symptom. On investigation, it turned out that the intern and nurse, who had worked together in bandaging him, had done so in such a way as to constrict certain blood vessels. The faulty bandaging had caused acute pain and retarded the patient's recovery. After the patient's eventual recovery, he sued the intern, claiming $20,000 in damages. Before the case went to trial, the patient and intern agreed to a settlement in which the intern paid the patient $12,000, and the patient released the intern from all further claims for liability arising out of the incident. If the intern brings suit against the nurse for contribution, the most likely result is that the intern will\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}