{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"587fde82-dab2-5166-9391-b00f86473bdc","task_key":"all--test--587fde82-dab2-5166-9391-b00f86473bdc","task_revision_id":"1","upstream_id":"","short_description":"A man and a woman worked for an armored car company, picking up and delivering…","config":"all","split":"test","body":"{\"choices\":\"[\\\"have an expert witness identify the man's signature as genuine.\\\", 'bring the original letter into court.', 'demonstrate the relevance of the letter.', 'present evidence of the authenticity of the letter.']\",\"question\":\"A man and a woman worked for an armored car company, picking up and delivering cash from banks and businesses. They figured out a plan to fake a holdup and robbery of the man's armored car. After faking the holdup, they planned to purchase a boat with the proceeds and travel to a remote island to deposit the money in a bank account there. The police got wind of the plan before it could be carried out, and the man and the woman were charged with conspiracy. At trial, the prosecution wants to introduce a letter from the man to the woman before the robbery in which the man wrote:\\\"Dear woman,I hope our plan for the heist works. If it does, we'll have a giant pile of bills to divide between us. (signed) the man\\\"Before the contents of the letter can be admitted into evidence, the prosecution must\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}