{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b684cc98-8755-5d44-995b-22f10503b580","task_key":"all--test--b684cc98-8755-5d44-995b-22f10503b580","task_revision_id":"1","upstream_id":"","short_description":"Mars has an atmosphere that is almost entirely carbon dioxide.Why isn't there a…","config":"all","split":"test","body":"{\"choices\":\"['the atmosphere on Mars is too thin to trap a significant amount of heat', 'There actually is a strong greenhouse effect and Mars would be 35oC colder than it is now without it.', 'Mars does not have enough internal heat to drive the greenhouse effect', 'the greenhouse effect requires an ozone layer which Mars does not have']\",\"question\":\"Mars has an atmosphere that is almost entirely carbon dioxide.Why isn't there a strong greenhouse effect keeping the planet warm?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}