{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6dd47799-0fe9-52ac-927a-aeb4d4e1d9bf","task_key":"all--test--6dd47799-0fe9-52ac-927a-aeb4d4e1d9bf","task_revision_id":"1","upstream_id":"","short_description":"A defendant and his friend were drinking heavily one night at the friend's…","config":"all","split":"test","body":"{\"choices\":\"['guilty, because the defendant was reckless.', 'guilty, because the defendant was criminally negligent.', 'not guilty, because the defendant did not commit a felony.', 'not guilty, because arson requires the presence of malice.']\",\"question\":\"A defendant and his friend were drinking heavily one night at the friend's house. They both became intoxicated. After several hours, both the friend and the defendant passed out. In the middle of the night, the defendant awoke with a headache. He went down to the kitchen and poured himself another shot of whiskey. The defendant also felt hungry and decided to make himself some eggs. After fixing his snack, he inadvertently forgot to turn off the stove. A gas leak occurred, causing a fire that burned down the house. If the defendant is charged with arson, he will be found\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}