{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7015e053-8b12-589d-94ed-dc3e0b4433d8","task_key":"all--test--7015e053-8b12-589d-94ed-dc3e0b4433d8","task_revision_id":"1","upstream_id":"","short_description":"Shortly after breaking up with the defendant, a woman began dating the victim.…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because federal conspiracy laws do not require the mental retainment ofjurisdictional requirements.', 'guilty, because federal conspiracy laws require only an intent to commit a prohibited act, but do not require a knowledge of the surrounding circumstances.', \\\"not guilty, because he didn't have the specific intent to commit the crime of assault in a federal park.\\\", 'not guilty, because he did not agree to commit a crime in a federal park.']\",\"question\":\"Shortly after breaking up with the defendant, a woman began dating the victim. The defendant, who still loved the woman, hated the victim. During a holiday weekend, the woman and the victim arranged to go camping in a federal park. The defendant and his friend decided to beat up the victim while he and the woman were on their camping trip. They went to the campsite where the woman and the victim were staying, but they couldn't find the couple, who were hiking in the woods. Subsequently, the defendant was arrested and charged with conspiracy to commit an assault in a federal park. At trial, the defendant testified that he didn't know he was in a federal park. Moreover, he stated that if he had known, he would never have agreed to the crime. If the jury believes the defendant, he should be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}