{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b1551af5-03c1-5c86-9340-562864b7267b","task_key":"default--test--1064","task_revision_id":"1","upstream_id":"1064","short_description":"A defendant was arrested and charged with arson, which in this jurisdiction is…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['proper, because it constituted a permissible inference.', \\\"proper, because the judge has discretion to guide the jury's deliberations.\\\", 'proper, because it constituted a rebuttable presumption of fact.', 'proper, because the law allows for the inference of intent based on circumstantial evidence.', 'improper, because the instruction assumes guilt based on possession of the matches.', 'improper, because the instruction shifts the burden of persuasion from the prosecution to the defendant defense.', 'proper, because it allowed the jury to consider all evidence presented.', 'improper, because it biased the jury against the defendant.', 'improper, because the prosecution must prove each element of the crime beyond a reasonable doubt.', \\\"improper, because it allowed the jury to speculate on the defendant's intent.\\\"]\",\"question\":\"A defendant was arrested and charged with arson, which in this jurisdiction is defined as the \\\"malicious or intentional burning of any structure or dwelling. \\\" When the defendant was arrested, he was inside a warehouse that was engulfed in flames. At the time he was apprehended, the police found in his possession a book of matches with four matches missing. At trial, the prosecution introduced the book of matches into evidence. Following closing arguments, and before the jury retired for deliberations, the judge gave instructions concerning the law of the case. The judge instructed the jury that it could infer the defendant's intent to set the fire based on the fact that four matches were missing from the matchbook. This instruction was specifically objected to by the defense counsel. The judge's jury instruction concerning the defendant's intent was\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}