{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"de9dc605-21df-5992-9334-9f543513515d","task_key":"all--test--de9dc605-21df-5992-9334-9f543513515d","task_revision_id":"1","upstream_id":"","short_description":"According to Ross, when our duties conflict with each other we should","config":"all","split":"test","body":"{\"choices\":\"['not do anything because duties cannot actually conflict with each other.', 'use moral judgment to determine which duty should be obeyed.', 'consult authorities instead of relying on our own judgment.', 'wait until the conflict ceases on its own.']\",\"question\":\" According to Ross, when our duties conflict with each other we should\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}