{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7354af5c-2c0e-5e01-add6-9b4cc3a5fc52","task_key":"all--test--7354af5c-2c0e-5e01-add6-9b4cc3a5fc52","task_revision_id":"1","upstream_id":"","short_description":"A company is considering a move to a Software-as-a-Service (SaaS) offering…","config":"all","split":"test","body":"{\"choices\":\"['Disaster recovery capabilities and documented recovery procedures.', 'User credential setup and control over the actions that employees can perform.', 'Allocation of software expenses and overhead charged to departments.', 'Ownership of processed data and costs of data migrations.']\",\"question\":\"A company is considering a move to a Software-as-a-Service (SaaS) offering instead of a traditional in-house application. Which of the following concerns is unique to SaaS?\",\"subject\":\"professional_accounting\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}