{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c02bfae2-d295-5a47-8aa6-c9f180c07aaf","task_key":"all--test--c02bfae2-d295-5a47-8aa6-c9f180c07aaf","task_revision_id":"1","upstream_id":"","short_description":"A federal statute established a national lottery and created a new federal…","config":"all","split":"test","body":"{\"choices\":\"['No, because a federal agency may not pay any monies to a state without explicit federal statutory authorization to do so.', 'No, because a state may not impose a tax directly on the federal government or any of its agencies or instrumentalities.', 'Yes, because it is a generally applicable tax that does not discriminate against the agency.', 'Yes, because the federal government is not exempt from state taxation on its commercial activities.']\",\"question\":\"A federal statute established a national lottery and created a new federal agency to administer it. In order to sell lottery tickets directly to the public, the agency established outlets throughout the country, including in a particular state. The state levies a general tax on the gross receipts from all gaming operations within the state. The state seeks to tax the agency on the sales of lottery tickets from its outlets within the state. Must the agency pay the state gross receipts tax?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}