{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a6121f3-0f2a-53f3-89a0-2fa2a050da1a","task_key":"all--test--7a6121f3-0f2a-53f3-89a0-2fa2a050da1a","task_revision_id":"1","upstream_id":"","short_description":"A man and a woman were passengers on a plane. They were seated next to one…","config":"all","split":"test","body":"{\"choices\":\"['admissible, because it is neither hearsay nor privileged.', 'inadmissible, because it is hearsay not within any recognized exception, though it is not privileged.', 'inadmissible, because it discloses a privileged communication, though it is not hearsay.', 'inadmissible, both because it discloses a privileged communication and because it is hearsay not within any recognized exception.']\",\"question\":\"A man and a woman were passengers on a plane. They were seated next to one another in the first- class section of the plane. Midway through the flight, the woman excused herself and went to the restroom. While she was gone, the man rummaged through the woman's purse and stole $100 in cash. Unknown to the man, a flight attendant saw him steal the money. As is customary in such situations, the flight attendant radioed ahead, and police officers arrested the man when the plane landed. The man was subsequently charged with violating an applicable federal larceny statute. During the trial, the prosecuting attorney called the man's wife as a willing witness against her husband. She proposed to testify that the man confided to her that he did steal the woman's money during the flight. The man's attorney objected on the grounds of hearsay and privilege. The wife's proposed testimony is\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}