{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"dad881a2-5766-5398-be65-30b673b5d924","task_key":"all--test--dad881a2-5766-5398-be65-30b673b5d924","task_revision_id":"1","upstream_id":"","short_description":"A woman was seven months pregnant with her first child. One afternoon, the woman…","config":"all","split":"test","body":"{\"choices\":\"['recover, because she suffered a physical injury, rather than just emotional harm.', 'recover, because the gang member acted intentionally in shooting the victim.', 'not recover, because the woman was not within the zone of danger.', 'not recover, because the gang member was unaware that the woman was watching.']\",\"question\":\"A woman was seven months pregnant with her first child. One afternoon, the woman was in the kitchen of her home preparing a snack when she glanced out the window and saw a horrible crime being committed. The woman watched in horror as a local gang member pulled out a handgun and used it to shoot the victim, a member of a rival gang. The gang member did not know anyone was witnessing his act of shooting the victim. The woman watched as the victim collapsed and fell dead on her neighbor's front lawn. As a result of her shock from viewing this horrible incident, the woman suffered a miscarriage. If the woman asserts a claim against the local gang member who shot the victim, the woman will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}