{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1b53e8e1-45ee-5b3e-b3a0-b32226b2123b","task_key":"default--test--1007","task_revision_id":"1","upstream_id":"1007","short_description":"A woman was married to a very wealthy man who had a generous life insurance…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['first-degree murder.', 'possession of a deadly substance.', 'second-degree murder.', 'assault with intent to kill.', 'manslaughter.', 'conspiracy to commit murder.', 'accessory to murder.', 'attempted murder.', 'aiding and abetting a felony.', 'no crime.']\",\"question\":\"A woman was married to a very wealthy man who had a generous life insurance policy. Deciding that she had had enough of married life, and desiring to spend more time with her boyfriend, she began to plan an early and unexpected demise for her husband. The woman devised a scheme whereby she would contaminate her husband's guacamole with a poisonous substance called strychnine. To facilitate her plan, she enlisted the help of her boyfriend. At first, the boyfriend adamantly refused to go along with her scheme, but when the woman threatened to break off their relationship, he reluctantly agreed. The boyfriend, a pre-med student, got the strychnine from the pharmacy by telling them that he was using it for an experiment with rats. The boyfriend then gave the strychnine to the woman, who sprinkled it in her husband's guacamole. The husband died from the poisoning. A statute in the jurisdiction provides: \\\"Any intentional and premeditated killing or one occurring during the perpetration of a felony, is murder in the first degree. Murder in the second degree is murder with intent to cause serious bodily harm. Manslaughter is an unlawful killing due to recklessness. \\\" Assault with intent to kill or to cause serious physical injury is a felony in the jurisdiction. There are no separate felonies relating to adulterating food or poisoning. The most serious crime for which the boyfriend can properly be convicted is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}