{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d59553b2-0409-55f4-9f21-3b0e56bf2eb3","task_key":"all--test--d59553b2-0409-55f4-9f21-3b0e56bf2eb3","task_revision_id":"1","upstream_id":"","short_description":"A man filed a federal diversity action against a bus company, seeking damages…","config":"all","split":"test","body":"{\"choices\":\"['No, because the court properly excused the three jurors due to illness.', 'No, because the parties stipulated to a verdict from a jury of fewer than six jurors.', 'Yes, because there must be at least six jurors on a federal civil jury.', 'Yes, because there must be at least 12 jurors on a federal civil jury.']\",\"question\":\"A man filed a federal diversity action against a bus company, seeking damages for injuries he had sustained in an accident while riding a bus owned by the company. The man demanded a jury trial. After the parties' attorneys examined the prospective jurors and exercised their challenges, six jurors and two alternate jurors were chosen. During the trial, two jurors became ill and were replaced by the alternate jurors. At the conclusion of replaced by the alternate jurors. At the conclusion of the trial, a third juror also became ill, and the court excused that juror. The parties' attorneys stipulated to the return of a verdict from a five-person jury. The jury then deliberated and returned a verdict for the company. The man timely filed a motion for a new trial, arguing that the five-person jury was not large enough to return a verdict. Should the court grant the motion?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}