{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d8906226-0084-5a41-b7f0-6cc1f58ba4e3","task_key":"all--test--d8906226-0084-5a41-b7f0-6cc1f58ba4e3","task_revision_id":"1","upstream_id":"","short_description":"According to de Marneffe, which of the following is a reason we might call…","config":"all","split":"test","body":"{\"choices\":\"['They use a drug to relieve a craving.', \\\"They use a drug even though they obviously shouldn't.\\\", 'Although they believe they should use the drug when they decide to use it, at other times they believe they should not use the drug anymore.', 'all of the above']\",\"question\":\" According to de Marneffe, which of the following is a reason we might call someone a drug addict?\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}