{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a4a4c81c-e605-51c6-9d7f-eaac72edd3ab","task_key":"default--test--1515","task_revision_id":"1","upstream_id":"1515","short_description":"At 10:00p. m. onNovember 14, a driver was operating his automobile along Main…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['not prevail, because the municipal water department would not be liable for the negligence of its independent contractor.', 'not prevail, because the driver was negligent for driving onto the sidewalk.', 'prevail, as the municipal water department is responsible for the safe operation of its contractors.', 'prevail, because the city government would be strictly liable for failing to ensure the water main repair work was done properly.', 'not prevail, as the municipal water department had no control over the actions of the driver or the other motorist.', 'not prevail, because sovereign immunity attaches to functions that are governmental in nature.', 'prevail, because the construction company failed to adequately warn the public of the hazard.', 'not prevail, because the accident was caused by the actions of another motorist, not the municipal water department.', 'prevail, because sovereign immunity would not attach to non-delegable duties, which are proprietary in nature.', 'prevail, because the municipal water department failed to ensure that the construction company had put adequate safety measures in place.']\",\"question\":\"At 10:00p. m. onNovember 14, a driver was operating his automobile along Main Street. As the driver was approaching the intersection of Main Street and First Avenue, a motorist, who was driving straight through a red light, suddenly appeared before him. Trying to avoid the motorist, the driver veered his car onto the sidewalk. The car landed in a deep hole in the sidewalk. This hole had been dug by a construction company, which had been repairing a water main break earlier in the day. The construction company had been hired by the local municipal water department. Although the' construction company had erected a warning sign advising pedestrians about the hole, there was no fence or barrier surrounding it. When the driver's car fell into the hole, it ruptured the water main, engulfing the car with water. Within a short time, the driver, unable to escape, drowned in his car, which rapidly filled with water. In a wrongful death action by the driver's estate against the municipal water department, the estate will most probably\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}