{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"36801d1b-d852-5103-8991-0caebb1f7955","task_key":"all--test--36801d1b-d852-5103-8991-0caebb1f7955","task_revision_id":"1","upstream_id":"","short_description":"You’ve made a scientific theory that there is an attractive force between all…","config":"all","split":"test","body":"{\"choices\":\"['The first time you drop a bowling ball and it falls to the ground proving your hypothesis.', 'After you’ve repeated your experiment many times.', 'You can never prove your theory to be correct only “yet to be proven wrong”.', 'When you and many others have tested the hypothesis.']\",\"question\":\"You’ve made a scientific theory that there is an attractive force between all objects. When will your theory be proven to be correct?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}