{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1f7e214e-6eae-5e02-b499-7d49561219af","task_key":"all--test--1f7e214e-6eae-5e02-b499-7d49561219af","task_revision_id":"1","upstream_id":"","short_description":"One evening, a defendant set fire to a homeowner's occupied house. As a result…","config":"all","split":"test","body":"{\"choices\":\"[\\\"granted, because the guilty verdict is plain error that adversely affects the defendant's constitutional rights.\\\", 'granted, because the verdicts are legally inconsistent and should lead to an acquittal of both charges.', 'denied, because the verdicts do not amount to a reversible error.', \\\"denied, because the defendant's proper remedy is to seek an appellate review for a non-constitutional error.\\\"]\",\"question\":\"One evening, a defendant set fire to a homeowner's occupied house. As a result of the blaze, the homeowner's daughter was killed. The defendant was charged with felony murder on the first count and arson on the second count of the two-count indictment. The jury found the defendant guilty on the first count, but returned a not guilty verdict on the second. The defendant's attorney's motion to set aside the guilty verdict on the felony murder charge will most likely be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}