{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f013d013-ed9f-59b5-bb00-f0123d74cee3","task_key":"default--test--1552","task_revision_id":"1","upstream_id":"1552","short_description":"A defendant was indicted and arrested for bank robbery. The defendant had an…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the defendant was not yet formally charged.', 'No.', 'Yes, based on the denial of counsel at both stages of the proceeding.', 'Yes, but only if the defendant specifically requested counsel at both stages.', 'No, because the Sixth Amendment right to counsel does not apply to initial appearances or lineups.', 'Yes, but only because the defendant was not provided a lawyer immediately after arrest.', 'Yes, but only if the defendant could not afford his own lawyer.', 'Yes, based only on the denial of counsel at the initial appearance.', 'No, because the defendant was provided counsel before the arraignment.', 'Yes, based only on the denial of counsel at the lineup.']\",\"question\":\"A defendant was indicted and arrested for bank robbery. The defendant had an initial appearance before a magistrate judge in which he was notified of the charges and told that counsel would be appointed for him the next day. The police then required the defendant to participate with other prisoners in a lineup in which each person had to wear a white T-shirt and say, \\\"Put your hands in the air.\\\" At the lineup, witnesses to the bank robbery identified the defendant as the bank robber. The next day, the defendant was arraigned on the charges. The defendant argues that his Sixth Amendment right to counsel was violated when he was denied counsel at two critical stages of the proceeding: his initial appearance in court before the magistrate initial appearance in court before the magistrate judge and the lineup identification. Was the defendant's Sixth Amendment right to counsel violated?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}