{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"af9600fb-7ff9-58b6-8b2a-5a2a959d2848","task_key":"all--test--af9600fb-7ff9-58b6-8b2a-5a2a959d2848","task_revision_id":"1","upstream_id":"","short_description":"A defendant is arrested and charged with shoplifting. He claims that he intended…","config":"all","split":"test","body":"{\"choices\":\"['No, because his actual sentence was only three months.', 'Yes, because the defendant was denied the right to counsel.', 'Yes, because the defendant was denied the right to a jury trial.', 'Yes, because the defendant was denied both the right to counsel and the right to a jury trial.']\",\"question\":\"A defendant is arrested and charged with shoplifting. He claims that he intended to pay for the item he took and requests an attorney and a jury trial. Both requests are denied by the judge, and the defendant is found guilty. The maximum penalty for shoplifting is a $500 fine and six months in jail. The defendant is sentenced to three months in jail, which he is allowed to serve on weekends in the county detention center. The defendant claims that his constitutional rights have been violated. Is he correct?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}