{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e89710ec-775f-531c-afea-32047b48832b","task_key":"default--test--952","task_revision_id":"1","upstream_id":"952","short_description":"A defendant was arrested and charged with burglarizing a pharmacy. The break-in…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because revealing the identity of the witness might put the witness at risk.', \\\"No, because it violates the defendant's right to a fair trial.\\\", 'No, because by requiring the defendant to reveal information before he is tactically ready to do so substantially impairs his ability to successfully defend himself.', 'Yes, because the prosecution should have an opportunity to investigate the merits of the alibi.', 'Yes, as long as the prosecution provides the defendant with a list of their witnesses in return.', 'Yes, because the defendant must prove his innocence.', 'Yes, because the court has a right to know the identity of all witnesses.', \\\"No, because such disclosure constitutes an implied representation that is testimonial in character and, thus, violates the defendant's privilege against self-incrimination.\\\", \\\"No, because the prosecution is not entitled to know the defense's strategy.\\\", 'Yes, because the defendant waived any claim of privilege when he sought discovery of the film from the prosecution.']\",\"question\":\"A defendant was arrested and charged with burglarizing a pharmacy. The break-in allegedly occurred late one evening after the store had closed for business. The defendant was identified as the perpetrator of the crime by a film that was recorded during the burglary from a hidden camera. When the defendant was apprehended, he denied involvement in the crime and told the arresting officers that he had been out of town when the burglary occurred. Prior to trial, the defendant's court appointed attorney filed a motion requesting discovery of the videotape film that was recorded during the perpetration of the crime. The trial judge granted the request and ordered the prosecution to send a duplicate copy of the videotape to the defendant's attorney. Following the judge's ruling, the prosecuting attorney, pursuant to state law, then filed a discovery motion specifically asking the defendant whether he planned to raise an alibi defense. The prosecuting attorney also sought discovery of the identity of such an alibi witness. Assume that the judge requires the defendant to respond as to whether the defendant intends to raise an alibi defense. May the judge also require the defendant to disclose the identity of the alibi witness?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}