{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d7cb7a75-0730-5f93-beae-78e0da66024b","task_key":"default--test--869","task_revision_id":"1","upstream_id":"869","short_description":"The defendant was walking down the street when he saw a woman struggling with a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Not knowing the true facts, the defendant was not justified in using force to protect the woman because the man was privileged to recapture his briefcase.', \\\"The defendant was justified in using any amount of force necessary in the woman's defense, since he believed she was in immediate danger.\\\", \\\"The defendant was not justified in using force, regardless of his belief about the situation, because the woman was attempting to steal the man's briefcase.\\\", 'The defendant was not justified in using force because he did not personally know the woman.', 'Since the defendant did not stand in any personal relationship with the woman, he was not justified in using force in her defense.', \\\"The defendant was justified in using force in the woman's defense, as he was acting in good faith and did not know the true circumstances.\\\", \\\"The defendant could use force in the woman's defense, provided that he did not cause more harm to the man than the woman was suffering.\\\", \\\"The defendant was justified in using reasonable force in the woman's defense, since his belief that she was in immediate danger of unlawful bodily harm from the man was both objectively and subjectively reasonable.\\\", \\\"The defendant was justified in using reasonable force in the woman's defense, since he reasonably believed she was in immediate danger of unlawful bodily harm from the man.\\\", 'The defendant was justified in using force, even if the woman was in the wrong, because he believed she was in danger.']\",\"question\":\"The defendant was walking down the street when he saw a woman struggling with a man over a briefcase. Unbeknownst to the defendant, the woman had just stolen the briefcase from the man on the street. Believing the woman to be the victim of an attack, the defendant intervened and punched the man until the woman was able to get away with the briefcase. Confused as to why he wasn't being hailed as a hero by the woman, the defendant eventually realized that he had been an unwitting accomplice to the woman's theft. The defendant apologized profusely to the man and went home. According to the alter ego rule, which of the following statements is correct with respect to the amount of force that the defendant was entitled to use in the woman's defense?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}