{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9da702af-429b-5dfa-aaa1-7b78ef357868","task_key":"default--test--1509","task_revision_id":"1","upstream_id":"1509","short_description":"A truck driver from State A and a bus driver from State B were involved in a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The court should apply the federal common law of negligence.', 'The court should consider the negligence law of both State A and State B and apply the law that the court believes most appropriately governs negligence in this action.', 'The court should apply the negligence law of the state with the highest population.', 'The court should apply international negligence law.', \\\"The court should apply the negligence law of the state where the truck driver's company is headquartered.\\\", \\\"The court should determine which state's negligence law a state court in State B would apply and apply that law in this action.\\\", \\\"The court should apply the negligence law of State A, the truck driver's state of citizenship.\\\"]\",\"question\":\"A truck driver from State A and a bus driver from State B were involved in a collision in State B that injured the truck driver. The truck driver filed a federal diversity action in State B based on negligence, seeking $100,000 in damages from the bus driver. What law of negligence should the court apply?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}