{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"430a9aac-ffc4-5850-a0e2-4ce610119aab","task_key":"all--test--430a9aac-ffc4-5850-a0e2-4ce610119aab","task_revision_id":"1","upstream_id":"","short_description":"A defendant was driving his new sports car at a high rate of speed on a busy…","config":"all","split":"test","body":"{\"choices\":\"['attempted murder.', 'attempted manslaughter.', 'assault with the intent to cause serious bodily injury.', 'battery.']\",\"question\":\"A defendant was driving his new sports car at a high rate of speed on a busy city street in the middle of the day. A pedestrian was struck by the defendant's car as she was crossing the street and was seriously injured. A criminal complaint was filed against the defendant and, at trial, he testified that he did not intend to injure anyone, but admitted that he was indifferent as to the consequences of driving recklessly. Based on the defendant's actions and testimony, he should be found guilty of\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}