{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"3aea2f3a-9fb2-5db6-a0c7-de5d8bf59264","task_key":"all--test--3aea2f3a-9fb2-5db6-a0c7-de5d8bf59264","task_revision_id":"1","upstream_id":"","short_description":"How might weapons belonging to state military or police forces arrive on the…","config":"all","split":"test","body":"{\"choices\":\"['States are strong enough to ensure that their weaponry is kept within their control and not traded illegally.', 'There is no evidence to suggest that state-owned weapons could be traded illicitly as part of the black or grey arms markets.', 'Illegal trading by states is the primary way in which weapons purchased legally might end up on the illegal arms market.', 'Looting and theft, especially where corruption is endemic, states are weak, and weapons stockpiles are not properly secured.']\",\"question\":\"How might weapons belonging to state military or police forces arrive on the illicit arms market?\",\"subject\":\"security_studies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}