{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c7d23d78-a96a-5dc5-b997-7ed89084ce49","task_key":"default--test--1812","task_revision_id":"1","upstream_id":"1812","short_description":"A homeowner was injured when an automatic cutoff switch failed to function on a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the manufacturer should have made the manual available to repair personnel.', 'Yes, because the snowblower should have had a backup safety system in place.', 'No, because the injury resulted from a substantial alteration of the snowblower by a third party. a third party.', 'No, because the mechanic is solely responsible for the improper repair.', \\\"No, because the homeowner was contributorily negligent in failing to furnish the snowblower's manual to the mechanic.\\\", 'No, because the homeowner should have ensured the mechanic had access to the correct manual.', \\\"Yes, because a defect in the snowblower caused the homeowner's injury.\\\", 'Yes, because the cutoff switch should not have failed regardless of the repair.', 'Yes, because the manufacturer should have made the switch mechanism more robust.', 'No, because the homeowner should have requested a professional repair from the manufacturer.']\",\"question\":\"A homeowner was injured when an automatic cutoff switch failed to function on a snowblower he was using. The cutoff switch had functioned well for a year after he purchased the snowblower but failed after the machine had been improperly repaired by a mechanic. The snowblower's operating manual contained a clear and prominent warning against making the very alteration to the switch mechanism that was made by the mechanic. The mechanic, however, did not have a manual available when he repaired the snowblower. Does the homeowner have a viable claim against the manufacturer of the snowblower for damages?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}