{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f9069f50-c1f2-5269-b235-03cea7a6de8a","task_key":"all--test--f9069f50-c1f2-5269-b235-03cea7a6de8a","task_revision_id":"1","upstream_id":"","short_description":"A gun dealer loaned a machine gun to a friend who told him only that he needed…","config":"all","split":"test","body":"{\"choices\":\"['Yes, he conspired to help the friend commit some unknown criminal act.', 'Yes, he is held to the foreseeability that the friend was going to commit a crime.', 'No, there was no agreement to conspire to rob a bank.', 'No, a conspiracy would have required that the gun dealer go to the bank with the friend.']\",\"question\":\"A gun dealer loaned a machine gun to a friend who told him only that he needed it to \\\"make some money.\\\" The friend was arrested for robbing a bank with the machine gun. The government charged the gun dealer and the friend with conspiracy to commit bank robbery. Can the charge of conspiracy to commit bank robbery be upheld against the gun dealer?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}