{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cbb7092f-17d9-5e22-9c44-95910cadcd95","task_key":"default--test--1681","task_revision_id":"1","upstream_id":"1681","short_description":"A security guard had a license to carry a concealed handgun. One day, he was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the security guard knew the barber was present.', 'not prevail, because the security guard did not physically harm the barber.', \\\"not prevail, because the barber was not a member of the friend's immediate family.\\\", 'not prevail, because the barber had a pre-existing health condition.', \\\"prevail, because the security guard's actions were intentionally frightening and harmful.\\\", 'prevail, because the security guard assaulted the friend.', \\\"prevail, because the security guard's actions were negligent and reckless.\\\", \\\"prevail, because the security guard's actions caused the barber severe emotional distress.\\\", 'not prevail, because the resulting harm was unforeseeable.', \\\"not prevail, because the security guard didn't directly target the barber.\\\"]\",\"question\":\"A security guard had a license to carry a concealed handgun. One day, he was walking past a barber shop when he saw his friend inside getting a haircut. The security guard knew that the barber, an elderly man, had a history of significant heart problems. The security guard decided to play a joke on his friend. The security guard took out his gun, burst through the door of the barber shop, pointed the gun at the friend, and shouted, \\\"Don't move or I'll shoot!\\\" The barber, thinking that an armed intruder was about to shoot the friend, suffered a heart attack and collapsed. If the barber brings suit against the security guard for infliction of emotional distress and seeks recovery for damages arising from the heart attack, he should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}