{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7269b7f0-b7df-5e8f-a2e2-162b5e2aefa5","task_key":"default--test--1871","task_revision_id":"1","upstream_id":"1871","short_description":"A wife was unhappily married to her husband, an alcoholic. When the husband got…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Yes, because, on these facts, a reasonable jury could conclude that the wife acted in self- defense by using necessary force to protect herself from the husband's constant violent attacks.\\\", \\\"Yes, because the wife was suffering from a psychological condition, battered women's syndrome, which justifies her actions.\\\", 'No, because the wife had the opportunity to leave the house but instead chose to kill her husband.', \\\"Yes, because the wife had no other means to protect herself from the husband's constant violent attacks.\\\", 'No, because even though the wife was the victim of spousal abuse, she could have avoided the danger by safely retreating.', \\\"No, because the wife had the option of seeking help from law enforcement or a women's shelter.\\\", 'Yes, because the wife was acting under duress due to constant physical abuse.', \\\"No, because the wife's belief in the necessity of deadly force in self-defense was unreasonable.\\\", 'No, because the wife premeditated the murder, therefore it cannot be considered self-defense.', \\\"Yes, because a criminal defendant's Sixth Amendment right to a jury trial prohibits a court from refusing to submit affirmative defenses to the jury.\\\"]\",\"question\":\"A wife was unhappily married to her husband, an alcoholic. When the husband got drunk, he would become abusive and beat the wife. During the course of their marriage, the beatings became more violent and more frequent. Unable to endure the physical abuse any longer, the wife hired a hit man to kill her husband. One night while the husband was asleep, the hit man entered the home with a key given to him by the wife and shot the husband to death. Afterward, the wife was prosecuted for murder as an accomplice. At trial, the prosecution presented the facts outlined above. Thereafter, the defense sought to have the judge instruct the jury that the wife acted in self- defense under the battered women's syndrome. Based on the given facts, should the judge give such an instruction to the jury?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}