{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e0122698-91a1-5b84-a9c5-70ebeb68202e","task_key":"default--test--926","task_revision_id":"1","upstream_id":"926","short_description":"A motorist was traveling at 5 miles per hour when she became distracted, and…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the motorist could not have predicted that such a small impact would have such severe consequences.', 'Yes, because the victim in this case is called a \\\"predictably unpredictable plaintiff.\\\"', \\\"No, because the passenger's rare medical condition was not directly caused by the motorist.\\\", 'Yes, because the motorist is always responsible for any harm caused by their negligent actions.', 'No, because the medical report did not conclusively prove that the collision was the sole cause of the disability.', 'Yes, because the law holds the defendant liable for all injuries, regardless of their severity or nature.', \\\"No, because the victim's psychosomatic reaction is usually held to be malingering under the law.\\\", 'Yes, because in the law of negligence the defendant takes the plaintiff \\\"as he finds her.\\\"', 'No, because the motorist is only liable for foreseeable injuries that are not of psychological origin.']\",\"question\":\"A motorist was traveling at 5 miles per hour when she became distracted, and negligently tapped the rear of another vehicle. A passenger in the struck vehicle had a rare medical condition that made her highly vulnerable to any slight physical impact or intrusion. The minimal impact caused the passenger to lightly bump her shoulder against the door, which triggered her rare immunosuppressant disease to surface. The passenger became permanently incapacitated from the impact. She sued the negligent motorist for all of her injuries. A physician submitted a report stating that the collision was a substantial factor in causing the disability. Has the plaintiff stated a claim for relief under prevailing legal principles?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}