{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a63e3c3f-61ab-5f51-946c-aaee9584ecb7","task_key":"all--test--a63e3c3f-61ab-5f51-946c-aaee9584ecb7","task_revision_id":"1","upstream_id":"","short_description":"A defendant was charged with illegally selling arms to a foreign country without…","config":"all","split":"test","body":"{\"choices\":\"['admissible, because it shows the absence of an entry in the record of a public agency.', \\\"admissible, because the official had firsthand knowledge of the information contained in the agency's employment records.\\\", 'inadmissible, because it is hearsay not within any recognized exception.', 'inadmissible, because the agency employment records are the best evidence to substantiate whether the defendant was actually in the employ of the governmental intelligence agency.']\",\"question\":\"A defendant was charged with illegally selling arms to a foreign country without a government license. During the trial, the defendant admitted selling the weapons, which included missiles, rocket launchers, bazookas, and ammunition, to agents of the foreign country. In his defense, however, the defendant testified that he did not sell the arms as a private citizen, but rather for his employer, a federal agency. In rebutting the defendant's testimony, the government called a high-ranking official from the federal agency to the stand. He testified that if the defendant were, in fact, a member of the agency, that information would ordinarily be contained in the agency's employment records. Asked whether the defendant's employment records were on file with the agency, the official then testified, \\\"I have searched the agency employment records diligently, but I haven't found any documentation that the defendant was employed by the agency. \\\"Upon objection by the defendant's attorney, the official's testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}