{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"102e58a1-38d7-599d-91cd-43a086c32e9a","task_key":"default--test--1169","task_revision_id":"1","upstream_id":"1169","short_description":"A traffic accident occurred at a road intersection. A motorcycle, a car, and a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['recover $5,000.', 'recover $45,000, the amount he paid to the motorcyclist.', 'recover $35,000.', 'recover $25,000.', 'recover nothing, as he has already collected from the motorcyclist.', 'recover $20,000.', 'recover nothing, because he was more at fault than the driver of the truck.', 'recover $27,000.', 'recover $15,000.', 'recover $30,000.']\",\"question\":\"A traffic accident occurred at a road intersection. A motorcycle, a car, and a truck were involved. The motorcyclist was injured and brought suit against the driver of the car and the driver of the truck. The jury returned a verdict finding that the motorcyclist's injuries were caused by negligence on the part of all three of the parties. The jury assigned 55 percent of the fault for the motorcyclist's injuries to the motorcyclist, 25 percent to the driver of the car, and 20 percent to the driver of the truck. The jury found that the amount of the motorcyclist's injuries was$100,000. The motorcyclist enforces the judgment against the driver of the car and collects $45,000 from the driver of the car. If the driver of the car then brings an action against the driver of the truck for contribution, the driver of the car should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}