{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"85768238-7640-5aad-a36e-24c287631df7","task_key":"all--test--85768238-7640-5aad-a36e-24c287631df7","task_revision_id":"1","upstream_id":"","short_description":"Two men were told by the defendant that a medical devices company was now using…","config":"all","split":"test","body":"{\"choices\":\"['an accessory before the fact.', 'a principal in the second degree.', 'an accomplice.', 'not respbnsible.']\",\"question\":\"Two men were told by the defendant that a medical devices company was now using the old warehouse on the river for storing its supply of defibrillators. Late one night, the two men broke into the warehouse and proceeded to load the large crates of defibrillators onto their truck. As they were leaving, one of the men inadvertently threw his cigarette butt into a refuse pile of old cardboard boxes and papers. Soon afterward, the refuse ignited into a small fire. Although the men had time to douse the fire without any danger to themselves, they did not do so. Instead, they fled. Meanwhile, the fire quickly spread, engulfing the entire warehouse in flames. If the defendant is later charged as a co-conspirator, in all likelihood he would be held responsible as\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}