{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"acdd25f8-5d89-5534-90bf-b4aad0d39ff7","task_key":"default--test--1908","task_revision_id":"1","upstream_id":"1908","short_description":"A pedestrian was injured when hit by a chair that was thrown from an upper-story…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the pedestrian has failed to offer evidence that the defendants jointly engaged in tortious conduct.', 'Yes, because the pedestrian has not provided evidence of negligence from a specific defendant.', 'Yes, because the defendants have not been proven guilty beyond a reasonable doubt.', 'No, because it is unreasonable to expect the pedestrian to prove which of the defendants caused the harm.', 'Yes, because the evidence provided is circumstantial and not directly linked to any defendant.', 'No, because of the doctrine of alternative liability.', 'No, because the pedestrian was not at fault for the incident.', 'No, because the defendants should prove their innocence.', 'No, because the pedestrian has suffered physical harm and someone must be held accountable.', \\\"Yes, because a plaintiff always has the burden to prove that a particular defendant's conduct was the factual cause of the plaintiff's physical harm.\\\"]\",\"question\":\"A pedestrian was injured when hit by a chair that was thrown from an upper-story hotel window. The pedestrian sued the occupants of all the rooms from which the chair might have been thrown. At trial, the pedestrian has been unable to offer any evidence as to the exact room from which the chair was thrown. The defendants have filed a motion for a directed verdict. Should the court grant the motion?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}