{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"cfd1d132-1ef4-5e9f-8108-1296e20063bf","task_key":"all--test--cfd1d132-1ef4-5e9f-8108-1296e20063bf","task_revision_id":"1","upstream_id":"","short_description":"Considering commerce and marketing, which of the following present the most…","config":"all","split":"test","body":"{\"choices\":\"['There is no direct return on investment in building security systems.', 'Security systems are detrimental to usability and can make IT systems less functional, and therefore less attractive to the consumer.', 'There is pressure to reduce the time it takes to get a new IT product or system onto the market, so security systems are sacrificed in order to reduce the time-to-market.', 'All of these options.']\",\"question\":\"Considering commerce and marketing, which of the following present the most significant obstacle to developing IT security?\",\"subject\":\"security_studies\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}