{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9a59b27f-b633-57a9-8738-f56a7c6b5315","task_key":"default--test--870","task_revision_id":"1","upstream_id":"870","short_description":"Two college students were members of the same fraternity. During a fraternity…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['guilty of assault, but not murder or manslaughter.', 'guilty of second-degree murder.', 'guilty of second-degree murder, due to the commission of a dangerous act.', 'guilty of first-degree murder.', 'guilty of manslaughter.', 'not guilty due to lack of intent.', 'not guilty, as he was misinformed about the gun being loaded.', 'guilty of first-degree murder, due to malice aforethought.', 'guilty of homicide, but not manslaughter or murder.', 'guilty of assault and manslaughter.']\",\"question\":\"Two college students were members of the same fraternity. During a fraternity party, the defendant pulled out a gun, pointed it at the other student and said, \\\"Beg for your life, sucker. \\\" The student, who knew that the defendant had a reputation as a practical joker, said, \\\"Get lost, you jerk. \\\" The defendant then pulled the trigger and shot the student to death. A statute in this jurisdiction provides: \\\"Any intentional and premeditated killing with malice aforethought is murder in the first degree. Murder in the second degree is any killing that occurs during the commission or attempted commission of a serious or inherently dangerous felony. Manslaughter includes all other types of unlawftil homicide and unjustifiable killings. \\\"The defendant was subsequently prosecuted for killing the student. At trial, the defendant testified that a fellow fraternity member gave him the gun before the shooting and told him that it was unloaded. If the jury believes the defendant, it should find him\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}