{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d2bc0f44-1b5f-5fa5-be2d-55abf9577214","task_key":"default--test--1322","task_revision_id":"1","upstream_id":"1322","short_description":"A longshoreman fell to his death through an open hatch on the deck of a ship.…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Deny the motion and submit the case to the jury with instructions that the ship's owner is liable if the hatch was left open irresponsibly.\\\", 'Grant the motion, because the custom should be considered conclusive on the issue of negligence.', 'Grant the motion, because it is customary for the crews to open the hatches for ventilation after the longshoremen have left the ship.', \\\"Deny the motion and submit the case to the jury with instructions that the ship's owner should win if the longshoreman was improperly near the hatch.\\\", 'Deny the motion, because the probability of serious injury caused by falling down an open hatch clearly outweighs the burden of keeping the hatch closed.', \\\"Grant the motion, as the longshoreman was an employee of a different company and the ship's owner can't be held responsible for his whereabouts.\\\", \\\"Deny the motion, because the ship's owner should have ensured that all hatches were closed after the longshoremen had left the ship.\\\", 'Grant the motion, because there was no reason for the longshoreman to have been near the hatch.', 'Deny the motion and submit the case to the jury with instructions that the custom is relevant but not conclusive on the issue of negligence.', 'Deny the motion and submit the case to the jury with instructions that the custom is irrelevant to the issue of negligence.']\",\"question\":\"A longshoreman fell to his death through an open hatch on the deck of a ship. The longshoreman was an employee of a company that had contracted with the ship's owner to load and unload the ship. The fall occurred at night, when loading work was over for the day, and there was no reason for the longshoreman to have been near the hatch. A negligence action was filed against the ship's owner for the death of the longshoreman. In that action, the owner has moved for summary judgment and has provided unrebutted evidence that it is customary for the crews of ships to open the hatches for ventilation after the longshoremen have left the ships. How should the court respond to the motion?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}