{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"da81f4dd-0307-5e38-918d-f3796b3a3a4d","task_key":"all--test--da81f4dd-0307-5e38-918d-f3796b3a3a4d","task_revision_id":"1","upstream_id":"","short_description":"A defendant was indicted and arrested for bank robbery. The defendant had an…","config":"all","split":"test","body":"{\"choices\":\"['No.', 'Yes, based only on the denial of counsel at the initial appearance.', 'Yes, based only on the denial of counsel at the lineup.', 'Yes, based on the denial of counsel at both stages of the proceeding.']\",\"question\":\"A defendant was indicted and arrested for bank robbery. The defendant had an initial appearance before a magistrate judge in which he was notified of the charges and told that counsel would be appointed for him the next day. The police then required the defendant to participate with other prisoners in a lineup in which each person had to wear a white T-shirt and say, \\\"Put your hands in the air.\\\" At the lineup, witnesses to the bank robbery identified the defendant as the bank robber. The next day, the defendant was arraigned on the charges. The defendant argues that his Sixth Amendment right to counsel was violated when he was denied counsel at two critical stages of the proceeding: his initial appearance in court before the magistrate initial appearance in court before the magistrate judge and the lineup identification. Was the defendant's Sixth Amendment right to counsel violated?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}