{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a5e0c198-ec2b-5599-9ec4-b045e1223b0a","task_key":"default--test--917","task_revision_id":"1","upstream_id":"917","short_description":"A shop owner domiciled in State A sued a distributor in a federal district court…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the distributor and the wholesaler conduct business in different states.', 'Yes, because there is no diversity of citizenship between the distributor and the wholesaler.', \\\"Yes, because the wholesaler's counterclaim is not directly related to the distributor's claim.\\\", \\\"No, because the wholesaler's and the distributor's principal places of business are diverse.\\\", 'No, because the distributor has invoked the jurisdiction of the court.', 'Yes, because the amount in controversy does not meet the jurisdictional limit.', \\\"Yes, because the wholesaler's counterclaim is not a compulsory counterclaim.\\\", 'No, because the wholesaler and distributor are incorporated in the same state.', \\\"No, because there is supplemental jurisdiction over the wholesaler's counterclaim.\\\", 'Yes, because there is no diversity of citizenship between the shop owner and the wholesaler.']\",\"question\":\"A shop owner domiciled in State A sued a distributor in a federal district court in State A for breach of a contract. The shop owner sought $100,000 in damages for allegedly defective goods that the distributor had provided under the contract. The distributor is incorporated in State B, with its principal place of business in State C. The distributor brought in as a third-party defendant the wholesaler that had provided the goods to the distributor, alleging that the wholesaler had a duty to indemnify the distributor for any damages recovered by the shop owner. The wholesaler is incorporated in State B, with its principal place of business in State A. The wholesaler has asserted a $60,000 counterclaim against the distributor for payment for the goods at issue, and the distributor has moved to dismiss the counterclaim for lack of subject-matter jurisdiction. Should the motion to dismiss be granted?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}