{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2baa3ad2-c7e9-583a-956e-313a6cdc6b30","task_key":"all--test--2baa3ad2-c7e9-583a-956e-313a6cdc6b30","task_revision_id":"1","upstream_id":"","short_description":"A defendant incurred heavy gambling losses to a bookie. Short of cash, the…","config":"all","split":"test","body":"{\"choices\":\"['an accessory before the fact.', 'a principal in the first degree.', 'a principal in the second degree.', 'an accomplice.']\",\"question\":\"A defendant incurred heavy gambling losses to a bookie. Short of cash, the defendant sought help from his uncle. When the uncle refused to lend his irresponsible nephew any money, the defendant decided to get even. One morning, as the housekeeper was preparing his uncle's breakfast, the defendant stealthily entered the kitchen and, unnoticed, dropped arsenic into the teapot from which his uncle was always served his tea. The defendant then hurriedly left the house and went to play golf. Shortly thereafter, the housekeeper served the uncle the poisoned tea, thus causing his unfortunate death. The defendant's culpability for his uncle's death would most probably be as\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}