{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bd96c527-1af0-578c-b088-1e60bd148e3c","task_key":"all--test--bd96c527-1af0-578c-b088-1e60bd148e3c","task_revision_id":"1","upstream_id":"","short_description":"Litigation arose in the courts of a state when the Secretary of Commerce…","config":"all","split":"test","body":"{\"choices\":\"['by appeal.', 'by certiorari.', 'if five justices vote to review it.', 'despite the doctrine of adequate and independent state grounds.']\",\"question\":\"Litigation arose in the courts of a state when the Secretary of Commerce attempted to stop a man from sending his new advertising brochures through the mail. Under the Secretary's interpretation, the new brochures contained obscene photographs and sexually suggestive language. The Secretary was acting under authority given to him by Section 1123(b) of state statute, which provided:\\\"Whenever the Secretary of Commerce shall determine that a commercial mailing to residents of the state is inappropriate for minors, the Secretary shall have the authority to prohibit such mailing. \\\"The case reached the highest court in the state, which held that the statute was unconstitutional, being in violation of both the commerce clause and the supremacy clause. If this case is reviewed by the U. S. Supreme Court, it will reach that court\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}