{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d1152cca-7cce-5cd2-93e0-941f4fa0df67","task_key":"default--test--1031","task_revision_id":"1","upstream_id":"1031","short_description":"A plaintiff sued a utility company that owns a reservoir that is open to the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['\\\"The plaintiff could have avoided the accident if he was more cautious.\\\"', '\\\"The plaintiff was not contributorily negligent.\\\"', '\\\"The accident probably occurred in the manner shown by this computer-animated film I made.\\\"', '\\\"The company\\\\'s negligence is a result of poor management.\\\"', '\\\"The company could have marked the island in a day and at a cost of $300.\\\"', '\\\"The company was required by federal law to mark the island.\\\"', '\\\"The federal agency is responsible for this accident.\\\"', '\\\"The company is responsible for all accidents in the reservoir.\\\"', '\\\"The company was not required by federal law to mark the island.\\\"']\",\"question\":\"A plaintiff sued a utility company that owns a reservoir that is open to the public for recreation pursuant to a license from a federal agency. The plaintiff was severely injured in the reservoir when he dove from a boat into what he thought was deep water and hit an unmarked submerged island. The plaintiff alleges that the company was negligent in failing to mark the submerged island. At trial, the plaintiff has called an engineer and qualified him as plaintiff has called an engineer and qualified him as an expert in managing reservoirs. Which of the following opinions by the plaintiff's expert is the court most likely to admit?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}