{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa3c185e-d182-528d-a509-a025e91cb401","task_key":"all--test--aa3c185e-d182-528d-a509-a025e91cb401","task_revision_id":"1","upstream_id":"","short_description":"Several persons together stole a painting from an art museum. One of them, who…","config":"all","split":"test","body":"{\"choices\":\"['Yes, as an admission by a party opponent.', 'Yes, as a statement against the interests of the other defendants.', 'No, because it was made while in custody and therefore is unreliable.', 'No, because the statement was not made in the course of or in furtherance of the conspiracy and thus it is hearsay.']\",\"question\":\"Several persons together stole a painting from an art museum. One of them, who was the driver, was caught during the commission of another crime. He told police that \\\"I didn't even get any money from the art museum. It all went to the first guy and the second guy.\\\" When the first and second guy go on trial for the theft of the art painting, is the driver's statement admissible in court?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}