{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7d7dbfdd-0cc9-500a-aa1b-6885453da03d","task_key":"default--test--1544","task_revision_id":"1","upstream_id":"1544","short_description":"A state enacted a statute that authorized the payment of state funds to any…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"uphold the validity of the statute as a valid exercise of the state's power to subsidize education.\\\", 'dismiss the action for lack of standing.', 'invalidate the statute on the grounds of separation of church and state.', \\\"invalidate the statute as a valid exercise of the state's power to subsidize education.\\\", 'uphold the validity of the statute but prohibit the subsidies to church-affiliated institutions.', 'invalidate the statute as violative of the establishment clause.', 'dismiss the action because the plaintiffs are not directly affected by the statute.', 'uphold the validity of the statute as violative of the establishment clause.', 'uphold the validity of the statute as nonviolative of the establishment clause.', 'dismiss the action on the basis that the statute does not promote a specific religion.']\",\"question\":\"A state enacted a statute that authorized the payment of state funds to any private institution of higher learning within the state that meets certain minimum criteria and refrains from awarding only seminarian or theological degrees. The aid is in the form of annual subsidies to qualified institutions. The grants are non-categorical but may not, under an added provision, be utilized by the institutions for sectarian purposes. A state agency determines the eligibility of applicant institutions. Eight state taxpayers, all avowed atheists, bring suit in federal court to enjoin the payments of subsidies under the statute to all church-affiliated institutions in the state. The federal district court will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}