{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"adb9583c-0033-5947-9df1-faf04d3baae8","task_key":"all--test--adb9583c-0033-5947-9df1-faf04d3baae8","task_revision_id":"1","upstream_id":"","short_description":"A defendant was at a bar when she saw her exboyfriend who had recently dumped…","config":"all","split":"test","body":"{\"choices\":\"['convicted, because her knowledge that her exboyfriend was a federal agent is not a material element of the crime.', 'convicted, because withdrawal is never recognized as an affirmative defense to conspiracy.', 'acquitted, because she was unaware that her ex-boyfriend was now a federal agent.', 'acquitted, because her abandonment of the plan aborted the conspiratorial objective.']\",\"question\":\"A defendant was at a bar when she saw her exboyfriend who had recently dumped her. The defendant walked away and spotted a good friend of hers. She approached her friend and told him she would give him $50 to beat up her ex-boyfriend. The friend agreed but said he would wait until the ex-boyfriend left the bar. While the defendant was eavesdropping on her ex-boyfriend in the hopes he would say something nice about her, she overheard him having a conversation on his cellphone. During the course of that conversation, the defendant realized that her ex-boyfriend was now a federal agent. Afraid she might be prosecuted for a federal offense, the defendant told her friend not to bother with the planned attack. If the defendant is subsequently charged with the statutory offense of conspiring to commit an assault on a federal officer, the defendant will probably be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}