{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3002a55c-27aa-5341-a781-c5093f0b8f79","task_key":"default--test--1785","task_revision_id":"1","upstream_id":"1785","short_description":"The defendant, who was walking on a public street, got into a heated argument…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, his use of a deadly weapon demonstrated the requisite intent to kill.', 'No, his behavior was reckless but not intentional.', 'No, the defendant was just trying to scare the victim, not kill him.', \\\"No, defendant was trying to protect himself from the victim's aggressive behavior.\\\", \\\"No, defendant was in a heated argument and his judgement was clouded, so he didn't have the intent to kill.\\\", 'Yes, as he should have known that using a deadly weapon could result in death, showing implied intent.', \\\"No, defendant responded to the victim's provocation and did not intend to kill.\\\", 'Yes, his use of a deadly weapon shows he had the intent to cause grievous harm, which resulted in death.', 'No, he acted on the spur of the moment and could not have formed the requisite intent.', 'Yes, because he intended to scare the victim with a deadly weapon, which is enough for a murder charge.']\",\"question\":\"The defendant, who was walking on a public street, got into a heated argument with a stranger who had been singing and shouting obscenities while walking near him. The defendant pulled a large serrated hunting knife and stabbed the victim in the chest, killing him. The authorities charged defendant with murder. The defendant argued at trial that he only intended to scare the victim to shut him up, and that he did not have the intent to kill. Is the defendant guilty of murder?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}