{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"89496d57-1999-5360-928b-c6f379595201","task_key":"all--test--89496d57-1999-5360-928b-c6f379595201","task_revision_id":"1","upstream_id":"","short_description":"An incumbent was running for re-election as the mayor of a city. During one of…","config":"all","split":"test","body":"{\"choices\":\"[\\\"admissible, because the television correspondent recorded the opponent's statement in her business capacity.\\\", \\\"admissible, because the television correspondent had firsthand knowledge of the opponent's statement.\\\", 'inadmissible, because the notebook entry is the best evidence.', 'inadmissible, because it is hearsay not within any recognized exception.']\",\"question\":\"An incumbent was running for re-election as the mayor of a city. During one of his opponent's campaign speeches, he stated that the mayor had once propositioned a prostitute. A television correspondent who was covering the election was present when the opponent made the statement. She recorded the opponent's remark about the mayor in her notebook. In a stunning upset, the opponent defeated the mayor in the election. Many of the pollsters attributed the mayor's loss to the question of his moral character following the opponent's smear campaign depicting the mayor as a client of prostitutes. Following the election, the mayor sued the opponent for defamation, basing his suit on the opponent's statement that the mayor had \\\"once propositioned a prostitute. \\\" At trial, the mayor calls the television correspondent to testify concerning her recollection of what the opponent said during the campaign speech attended by the television correspondent. The television correspondent's testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}