{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"66bcef76-d5b6-5f89-be0c-75a54be47751","task_key":"all--test--66bcef76-d5b6-5f89-be0c-75a54be47751","task_revision_id":"1","upstream_id":"","short_description":"Which of the following correctly states the relationship between the federal and…","config":"all","split":"test","body":"{\"choices\":\"['Federal courts are higher courts than state courts and may overturn state decisions on any grounds.', 'The two are entirely autonomous, and neither ever hears cases that originate in the other.', 'The two are generally autonomous, although federal courts may rule on the constitutionality of state court decisions.', 'State courts are trial courts; federal courts are appeals courts.']\",\"question\":\"Which of the following correctly states the relationship between the federal and state judiciaries?\",\"subject\":\"high_school_government_and_politics\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}