{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"55b493bb-5c5d-53a4-9ebe-1794fdca13ec","task_key":"all--test--55b493bb-5c5d-53a4-9ebe-1794fdca13ec","task_revision_id":"1","upstream_id":"","short_description":"An entrepreneur from State A decided to sell hot sauce to the public, labeling…","config":"all","split":"test","body":"{\"choices\":\"[\\\"No, because the company's claim arises under federal law.\\\", 'No, because the entrepreneur waived the right to chal-lenge subject-matter jurisdiction by not raising the issue initially by motion or in the answer.', 'Yes, because although the claim arises under federal law, the amount in controversy is not satisfied.', 'Yes, because although there is diversity, the amount in controversy is not satisfied.']\",\"question\":\"An entrepreneur from State A decided to sell hot sauce to the public, labeling it ÒBest Hot Sauce.Ó A company incorporated in State B and headquartered in State C sued the entrepreneur in federal court in State C. The complaint sought $50,000 in damages and alleged that the entrepreneur's use of the name ÒBest Hot SauceÓ infringed the company's federal trademark. The entrepreneur filed an answer denying the allegations, and the parties began discovery. Six months later, the entrepreneur moved to dismiss for lack of subject-matter jurisdiction. Should the court grant the entrepreneur's motion?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}