{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b525a8b9-bc3a-57d2-b0d5-375b9da4483a","task_key":"all--test--b525a8b9-bc3a-57d2-b0d5-375b9da4483a","task_revision_id":"1","upstream_id":"","short_description":"According to the Solar Nebular theory what are asteroids and comets?","config":"all","split":"test","body":"{\"choices\":\"['They are the shattered remains of collisions between planets.', 'They are chunks of rock or ice that condensed long after the planets and moons had formed.', 'They are chunks of rock or ice that were expelled from planets by volcanoes.', 'They are leftover planetesimals that never accreted into planets.']\",\"question\":\"According to the Solar Nebular theory what are asteroids and comets?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}