{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e72c2b4b-067c-5447-a922-ff73ece75323","task_key":"all--test--e72c2b4b-067c-5447-a922-ff73ece75323","task_revision_id":"1","upstream_id":"","short_description":"The astronomical unit parsec (pc) plays a crucial role in astronomy. One parsec…","config":"all","split":"test","body":"{\"choices\":\"['Distance at which one astronomical unit measures one arcsecond from Earth.', 'Orbital distance of the solar system around the center of the Milky Way in one year.', 'Effective distance of the solar wind (i.e. the radius of the heliosphere).', 'Historical distance to the brightest star Sirius.']\",\"question\":\"The astronomical unit parsec (pc) plays a crucial role in astronomy. One parsec is equal to about 3.26 light-years. How is one parsec defined in astronomy?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}