{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"34721058-f5da-52fa-825a-1300e5875b4c","task_key":"all--test--34721058-f5da-52fa-825a-1300e5875b4c","task_revision_id":"1","upstream_id":"","short_description":"You've made a scientific theory that there is a force between all objects. This…","config":"all","split":"test","body":"{\"choices\":\"['When you and many other Jedi have tested the hypothesis', 'Jean-Luc Skywalker drops Yoda during the day and he falls to the ground proving your theory.', 'You can never prove your theory to be correct only \\\"yet to be proven wrong.\\\"', \\\"After you've repeated your experiment many times\\\"]\",\"question\":\"You've made a scientific theory that there is a force between all objects. This forcce has both a dark and light side. When will your theory be proven correct?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}