{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"fd68bfee-0084-50c1-8475-97b25b8e8c77","task_key":"default--test--1132","task_revision_id":"1","upstream_id":"1132","short_description":"There was an accident at a manufacturing plant located in California which state…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Apply the federal punitive damage rule since the case is in a federal court.', 'Apply the punitive damage rule in California since that is where the manufacturing plant is located.', 'Apply the punitive damage rule in Washington since that is where the lawsuit was filed.', 'Apply the punitive damage rule in California since that is where the accident occurred.', 'Let the jury in the case decide which state law of punitive damage should apply based upon their analysis of the relative culpability of the parties.', 'Apply the punitive damage rule in Washington because that is where the court is located.', 'Apply the punitive damage rule in Washington since that is where the plaintiff resides.', 'Apply the punitive damage rule in the state that has the most favorable law for the defendant.', 'Apply the punitive damage rule in the state that has the most favorable law for the plaintiff.', \\\"Apply the punitive damage rule of the state where the company's headquarters is located.\\\"]\",\"question\":\"There was an accident at a manufacturing plant located in California which state generally allows prevailing parties to collect punitive damages. A worker from Washington was injured at the California plant and she filed suit in federal district court in her home state of Washington. Washington does not generally allow punitive damages. The plaintiff has alleged in their federal lawsuit filed in Washington that they are entitled to punitive damages. The court should\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}