{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"846e7378-3929-50da-87b4-ede5a39dc0b6","task_key":"all--test--846e7378-3929-50da-87b4-ede5a39dc0b6","task_revision_id":"1","upstream_id":"","short_description":"A heavyset man with long red hair robbed a liquor store. Thereafter, a man was…","config":"all","split":"test","body":"{\"choices\":\"['admissible as a prior identification by the witness.', 'admissible as past recollection recorded.', 'inadmissible, because it is hearsay not within any recognized exception.', \\\"inadmissible, because it is a violation of the man's right of confrontation.\\\"]\",\"question\":\"A heavyset man with long red hair robbed a liquor store. Thereafter, a man was arrested and charged with the armed robbery. At the man's trial, the owner of the liquor store was called to testify. He admitted that he was unable to identify the man, who now had a shaven head, as the robber. The prosecuting attorney then handed the owner six photographs. He proposed to testify, over defense objections, that he had previously told the prosecuting attorney that picture #4, admittedly a picture of the man before he shaved his head, was a picture of the person who robbed his store. The owner's proffered testimony should be adjudged\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}