{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e2474635-665c-5440-8adb-38054dd9ff52","task_key":"all--test--e2474635-665c-5440-8adb-38054dd9ff52","task_revision_id":"1","upstream_id":"","short_description":"In response to the objection that it would be wrong to prohibit the manufacture…","config":"all","split":"test","body":"{\"choices\":\"['fatty foods and tobacco products are not actually harmful.', \\\"there is nothing wrong with limiting people's access to products that are harmful to them.\\\", 'the benefits of prohibiting these products would not outweigh the costs.', 'all of the above']\",\"question\":\" In response to the objection that it would be wrong to prohibit the manufacture and sale of fatty foods and tobacco products, de Marneffe argues that\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}