{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1e3e6d41-1623-577a-b875-0b759fab9bbe","task_key":"all--test--1e3e6d41-1623-577a-b875-0b759fab9bbe","task_revision_id":"1","upstream_id":"","short_description":"Two weeks before an election, a local newspaper published an article implying…","config":"all","split":"test","body":"{\"choices\":\"['excluded, because bad acts may not be proved by extrinsic evidence.', 'excluded, because it is improper character evidence.', \\\"admitted as probative evidence of the candidate's character for veracity.\\\", \\\"admitted as relevant evidence of the candidate's character as a thief.\\\"]\",\"question\":\"Two weeks before an election, a local newspaper published an article implying that one of the mayoral candidates was a thief. The candidate subsequently lost the election and sued the newspaper for defamation. The newspaper defended on the grounds of truth. At trial, the candidate took the stand and testified that he was not a thief. Thereafter, the newspaper called a witness to testify that three years ago the candidate committed a larceny while employed at his former job. Upon objection by the candidate's attorney, the witness's testimony should be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}