{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"49862ad7-c2c3-5734-ad9c-e954824f779c","task_key":"default--test--1786","task_revision_id":"1","upstream_id":"1786","short_description":"In a trial of a defendant for armed bank robbery, the prosecutor presented…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"Affirm the conviction, because the defense attorney failed to timely object to the judge's action.\\\", \\\"Affirm the conviction, because the defense attorney's objection was not valid.\\\", 'Reverse the conviction, because the jury did not have the opportunity to deliberate freely.', 'Reverse the conviction, because the fingerprint evidence was not properly handled by the police.', \\\"Reverse the conviction, because the judge's action in directing the verdict denied the defendant his constitutional right to a trial by jury.\\\", \\\"Affirm the conviction, because the prosecution presented both circumstantial and direct evidence conclusively establishing the defendant's guilt.\\\", 'Affirm the conviction, because the defendant was caught immediately after the crime.', \\\"Affirm the conviction, because the judge's action in directing the verdict is within his rights when evidence is overwhelming.\\\", 'Reverse the conviction, because the defendant was not given an opportunity to present a defense.', \\\"Reverse the conviction, because the defense attorney's failure to timely object to the judge's action constituted ineffective assistance of counsel.\\\"]\",\"question\":\"In a trial of a defendant for armed bank robbery, the prosecutor presented overwhelming evidence of guilt. Three tellers identified the defendant as the robber, a latent fingerprint found on the bank counter linked the defendant to the scene, and bank money had been found in the defendant's car. The police had arrested the defendant immediately after the robbery, as the defendant was driving away from the bank. the bank. When the prosecution rested its case, the judge directed the jury to return a verdict of guilty. The defense attorney did not object at the time. The jury withdrew, discussed the case, and agreed to a guilty verdict. The jury returned to the courtroom and announced the guilty verdict. The defense attorney then voiced an objection to the judge's having directed the verdict. The court overruled the objection and sentenced the defendant to 20 years in prison. On appeal, what should the appellate court do?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}