{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5b26e298-66b9-52a2-bb3e-671b4a405892","task_key":"all--test--5b26e298-66b9-52a2-bb3e-671b4a405892","task_revision_id":"1","upstream_id":"","short_description":"The climate of Great Britain is milder than most other places along the same…","config":"all","split":"test","body":"{\"choices\":\"['The Gulf Stream brings warm waters to Great Britain, raising the temperature of winds that blow onto the island.', 'Hot springs throughout the country raise the temperature of the surrounding land and air.', 'Mountain ranges in the north of Great Britain act as a barrier to cold winds blowing from the Arctic.', 'Small landmasses generate their own local climate and are largely unaffected by latitude.']\",\"question\":\"The climate of Great Britain is milder than most other places along the same latitude for which of the following reasons?\",\"subject\":\"high_school_geography\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}