{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0d35ea91-8108-5c1a-877d-9613bd8fa36c","task_key":"default--test--1791","task_revision_id":"1","upstream_id":"1791","short_description":"A farmer purchased a new pickup truck from a local automobile dealership. A few…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the pedestrian was not a purchaser or user of the truck.', 'Yes, if the manufacturing company had issued a recall for the defect.', 'No, if the farmer was speeding and therefore could not stop in time.', 'Yes, if the defect could have been discovered through the exercise of reasonable care by the auto dealer.', 'No, if the pedestrian was crossing the street illegally.', 'No, if the farmer was negligent in being unable to steer around and avoid hitting the pedestrian.', 'Yes, if the auto dealer was aware of the defect but did not inform the farmer.', 'No, because the farmer should have had the truck inspected before using it for heavy duty work.']\",\"question\":\"A farmer purchased a new pickup truck from a local automobile dealership. A few weeks later, the farmer was using the truck to haul a load of supplies to his farm. As he drove down a small hill toward an intersection with a stop sign, he applied the brakes, but they failed to work, and the truck did not slow down. The farmer could not stop for the stop sign. He saw a pedestrian crossing the street at the intersection and tried to steer around him, but was unable to do so. The pickup truck struck the pedestrian and seriously injured him. If the injured pedestrian asserts a negligence claim against the auto dealership that sold the truck to the farmer, and if it is conclusively proven that the brake failure resulted from a manufacturing defect in the car, will the pedestrian prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}