{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d91f5030-4487-5493-9948-bc2ef756a81b","task_key":"default--test--1530","task_revision_id":"1","upstream_id":"1530","short_description":"An entrepreneur from State A decided to sell hot sauce to the public, labeling…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because although there is diversity, the amount in controversy is not satisfied.', 'No, because the entrepreneur waived the right to chal-lenge subject-matter jurisdiction by not raising the issue initially by motion or in the answer.', \\\"No, because the company's headquarters are in the same state as the court.\\\", 'Yes, because the complaint is about a product, not a legal matter.', \\\"No, because the entrepreneur's product is being sold to the public.\\\", 'No, because the entrepreneur has been participating in the case for six months.', 'Yes, because the entrepreneur is not from the same state as the company.', 'Yes, because the entrepreneur is not a resident of the state where the court is located.', \\\"No, because the company's claim arises under federal law.\\\", 'Yes, because although the claim arises under federal law, the amount in controversy is not satisfied.']\",\"question\":\"An entrepreneur from State A decided to sell hot sauce to the public, labeling it ÒBest Hot Sauce.Ó A company incorporated in State B and headquartered in State C sued the entrepreneur in federal court in State C. The complaint sought $50,000 in damages and alleged that the entrepreneur's use of the name ÒBest Hot SauceÓ infringed the company's federal trademark. The entrepreneur filed an answer denying the allegations, and the parties began discovery. Six months later, the entrepreneur moved to dismiss for lack of subject-matter jurisdiction. Should the court grant the entrepreneur's motion?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}