{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fe2ee752-a159-5533-b7fb-05fa8d050378","task_key":"all--test--fe2ee752-a159-5533-b7fb-05fa8d050378","task_revision_id":"1","upstream_id":"","short_description":"To say that an action is intrinsically permissible (a feature mentioned in the…","config":"all","split":"test","body":"{\"choices\":\"['the action, apart from its effects, is morally permissible.', 'the action, only because of its effects, is morally permissible.', 'no actual person would ever endorse that action.', 'the action is valuable because of its relationship to things that are intrinsically valuable.']\",\"question\":\" To say that an action is intrinsically permissible (a feature mentioned in the doctrine of double effect) is to say that\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}