{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"be23e6f4-3170-5d9c-bc70-426701b13583","task_key":"all--test--be23e6f4-3170-5d9c-bc70-426701b13583","task_revision_id":"1","upstream_id":"","short_description":"Some of Mars'water is frozen in the soil and the ice caps and some was lost due…","config":"all","split":"test","body":"{\"choices\":\"['it was stripped away by the magnetic field when it rapidly decreased in strength', 'it was broken into hydrogen and oxygen by ultraviolet light and the hydrogen was lost through thermal escape', 'it was consumed by a civilization of thirsty Martians', 'it was vaporized during a period of intense volcanism and was lost to space through thermal escape']\",\"question\":\"Some of Mars'water is frozen in the soil and the ice caps and some was lost due to solar wind stripping but much of the original water is thought to have been lost through another mechanism. What happened to this \\\"lost\\\"water?\",\"subject\":\"astronomy\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}