{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8f13b681-7564-5aee-a573-efc016396831","task_key":"default--test--1877","task_revision_id":"1","upstream_id":"1877","short_description":"A repairman repaired damaged electrical power lines and replaced old, worn-out…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the pedestrian was acting as a good Samaritan.', 'prevail, because the company should have ensured the area was safe for pedestrians.', 'not prevail, because the company had no control over the ladder slipping due to wet ground.', 'prevail, because he was attempting to rescue the repairman.', 'prevail, because the company failed to provide immediate assistance to the repairman.', 'not prevail, because the electrical company is not liable for accidents caused by natural disasters.', 'prevail, because a public utility company is strictly liable in tort.', 'not prevail, because the repairman was not negligent in not affixing the ladder properly.']\",\"question\":\"A repairman repaired damaged electrical power lines and replaced old, worn-out equipment whenever necessary for the local municipal electrical company. After a violent tornado had ripped through the city, the electrical company was busily trying to restore electrical power to its customers. The morning after the storm, the repairman was perched on a ladder trying to repair a high-voltage power line. As he was removing the cover of the transformer box, the ladder suddenly slipped on the wet ground, even though it had been properly fixed into position. The repairman struggled to maintain his balance and, luckily, the ladder came to rest against the transformer box. A pedestrian, who was walking on the street below, saw the repairman's predicament and began to climb the ladder to aid him. As the pedestrian was ascending the ladder, his foot slipped, and he fell to the ground. His fall caused the ladder to jar loose the transformer box, which in turn sent the repairman falling to the pavement. The repairman and the pedestrian both suffered serious injuries. If the pedestrian asserts a claim against the electrical company for his injuries, he will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}