{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5a675f77-65bc-50b4-a464-85fbf97768ed","task_key":"all--test--5a675f77-65bc-50b4-a464-85fbf97768ed","task_revision_id":"1","upstream_id":"","short_description":"After waiting in line for two hours to gain entry into a popular nightclub, a…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because they were killings that occurred during the commission of an inherently dangerous felony.', 'guilty, because duress is not a defense to murder.', 'not guilty, because duress is a defense to arson.', 'not guilty, because the defendant was justified under the circumstances.']\",\"question\":\"After waiting in line for two hours to gain entry into a popular nightclub, a man was denied admission because his attire failed to conform to the club's dress code. When he was refused admittance, the man angrily shouted to the club's doorman, \\\"You'll be sorry for this. After I'm through, this club will be reduced to rubble. \\\" Later that same evening, the man returned to the club with two explosives in his possession. He noticed the defendant leaving the club and followed him into a nearby parking lot. As the defendant was about to enter his car, the man grabbed him, pointed a gun and said, \\\"Follow me, you punk, or I'll blow your brains out. \\\" The man led the defendant to the rear of the club, handed him the explosives, and directed him to throw them through an open window of the club. In fear of his life, the defendant tossed the explosives into the club, causing an inferno that killed 25 people. In this jurisdiction, the applicable arson statute includes all buildings. If the defendant is charged with felony murder for the death of the people in the club, he will most likely be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}