{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"63a0c3f6-ac56-5b23-a3f2-908cebbda558","task_key":"default--test--1095","task_revision_id":"1","upstream_id":"1095","short_description":"A buyer purchased a new van from a car dealer. Two weeks later, the buyer was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['denied, because the buyer should have noticed the defect within two weeks of purchasing the van.', 'denied, because the buyer is strictly liable for injuries caused by a manufacturing defect in an auto that he had purchased.', 'granted, because the woman was neither the user nor the consumer of the defective product.', 'granted, because the woman presented no evidence that the buyer was negligent.', 'granted, because the buyer was not responsible for the manufacturing defect.', 'granted, because the woman failed to prove the buyer had knowledge of the defect.', 'denied, because the buyer had a non-delegable duty to maintain the brakes in a safe condition.', 'denied, because the buyer should have performed a thorough check of the van after purchase.', 'granted, because the woman did not present evidence that the accident was directly caused by the brake failure.', 'denied, because the buyer did not return the van upon noticing the brake issue.']\",\"question\":\"A buyer purchased a new van from a car dealer. Two weeks later, the buyer was driving to work when the brakes suddenly failed. The buyer tried to stop the van for a red light, but the brakes failed to operate. As a consequence, the buyer drove through the red light and collided with a car driven by a woman. Subsequently, the woman asserted a claim against the buyer to recover for the injuries she suffered in the accident. At trial, the only evidence offered by the plaintiff concerning the cause of the accident was the testimony of an engineering expert. He testified that a manufacturing defect had caused the brakes to suddenly fail. Based on the facts stated above, a motion by the buyer to dismiss at the end of the woman's case should be\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}