{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"377934b7-8533-5b15-b90d-839797dcabf2","task_key":"all--test--377934b7-8533-5b15-b90d-839797dcabf2","task_revision_id":"1","upstream_id":"","short_description":"Suppose two athletes, A and B, are equally skilled. Suppose also that A…","config":"all","split":"test","body":"{\"choices\":\"['A is a better athlete.', 'B is a better athlete.', \\\"B doesn't count as an athlete because training precludes athleticism.\\\", 'A and B are equally good athletes.']\",\"question\":\" Suppose two athletes, A and B, are equally skilled. Suppose also that A naturally inherited certain traits from her father that made playing the sport easier for her, and thus she didn't have to train as hard as B. According to the ethic of willfulness,\",\"subject\":\"moral_disputes\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}