{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dae65cb4-4b6d-50d9-ac72-0b1ada3dd865","task_key":"all--test--dae65cb4-4b6d-50d9-ac72-0b1ada3dd865","task_revision_id":"1","upstream_id":"","short_description":"After a wife found out about her husband's infidelity, she decided to have him…","config":"all","split":"test","body":"{\"choices\":\"['guilty of solicitation.', 'guilty of conspiracy to commit murder.', 'guilty of solicitation and conspiracy to commit murder.', 'not guilty of either solicitation or conspiracy to commit murder.']\",\"question\":\"After a wife found out about her husband's infidelity, she decided to have him killed. The defendant approached a hit man to solicit his assistance in carrying out the planned murder. Although the wife believed that he was a hit man, he was, in fact, an undercover agent. The wife told the undercovr agent she would pay him $20,000 if he accepted the job and an additional $20,000 after the killing was completed. Upon hearing the wife's proposal, the undercover agent told the wife he would kill her husband. However, the undercover agent secretly intended not to go through with the plan, and he merely feigned agreement because he wished to trap the wife. The wife told the undercover agent that she would deliver the first payment the next day. However, before making the initial payment, the wife learned that the hit man was really an undercover agent. Fearful that she might be prosecuted for planning her husband's murder, the wife contacted the police and renounced her participation in the criminal endeavor. The defendant will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}