{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b0e55f23-1790-558e-885b-d5df5d6bcc5d","task_key":"all--test--b0e55f23-1790-558e-885b-d5df5d6bcc5d","task_revision_id":"1","upstream_id":"","short_description":"A plaintiff has sued a defendant in a court of the state in which both parties…","config":"all","split":"test","body":"{\"choices\":\"['No, because cases arising under federal law must be decided in federal court.', 'No, because state courts must abstain in cases arising under federal law until a federal court has decided the federal issue.', 'Yes, because state courts may not discriminate against cases arising under discriminate against cases arising under federal law.', 'Yes, because the parties cannot proceed in federal court since there is no diversity of citizenship.']\",\"question\":\"A plaintiff has sued a defendant in a court of the state in which both parties reside. The plaintiff alleges only a cause of action arising under a federal statute, although state law provides a similar cause of action. The federal statute provides that claims under the statute can be brought in any court of competent jurisdiction. The statute has not yet been interpreted by any federal court. Should the state court hear the case?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}