{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3d6cd27e-cad2-5f23-9724-c5348384abce","task_key":"all--test--3d6cd27e-cad2-5f23-9724-c5348384abce","task_revision_id":"1","upstream_id":"","short_description":"A defendant was charged with felony murder as a result of his setting fire to…","config":"all","split":"test","body":"{\"choices\":\"['The defendant did not intend to kill the victim.', 'The defendant was insane when he set the fire.', 'The defendant was coerced by another to set the fire.', 'The victim died of a heart attack before the fire spread to his bedroom.']\",\"question\":\"A defendant was charged with felony murder as a result of his setting fire to the victim's house. The victim was upstairs sleeping when the fire was set, and he died in his bedroom during the fire. If the defendant can prove the facts to support his defense, which of the following assertions would LEAST likely remove liability for felony murder?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}