{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"27708611-e501-57fa-a3e6-4ac67577da74","task_key":"all--test--27708611-e501-57fa-a3e6-4ac67577da74","task_revision_id":"1","upstream_id":"","short_description":"An elderly woman was mugged and robbed on a public street. At the time of the…","config":"all","split":"test","body":"{\"choices\":\"['No, because it is not reliable enough and there was too much suggestion to make a reasonably accurate identification.', 'No, because prior statements of identification are considered hearsay and never allowed into evidence.', 'Yes, if the witness perceived the perpetrator by seeing his eyes, and made the identification on that basis it was reliable enough to be admitted.', \\\"Yes, because the witness' recognition of someone is all that is needed; the reliability or credibility of it is subject to cross examination.\\\"]\",\"question\":\"An elderly woman was mugged and robbed on a public street. At the time of the offense, the victim only saw her assailant's eyes. A hooded sweatshirt and bandana concealed the rest of his face. A few days later, while the victim waited to be interviewed, she observed the defendant being escorted through the courthouse, in jail clothing, and restrained by handcuffs and leg irons. She identified him as the perpetrator shortly thereafter. Will the court allow that identification into evidence?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}