{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"527f0a0e-e25f-533f-9b76-823a3995a63a","task_key":"all--test--527f0a0e-e25f-533f-9b76-823a3995a63a","task_revision_id":"1","upstream_id":"","short_description":"A plaintiff sued a utility company that owns a reservoir that is open to the…","config":"all","split":"test","body":"{\"choices\":\"['\\\"The accident probably occurred in the manner shown by this computer-animated film I made.\\\"', '\\\"The company could have marked the island in a day and at a cost of $300.\\\"', '\\\"The company was required by federal law to mark the island.\\\"', '\\\"The plaintiff was not contributorily negligent.\\\"']\",\"question\":\"A plaintiff sued a utility company that owns a reservoir that is open to the public for recreation pursuant to a license from a federal agency. The plaintiff was severely injured in the reservoir when he dove from a boat into what he thought was deep water and hit an unmarked submerged island. The plaintiff alleges that the company was negligent in failing to mark the submerged island. At trial, the plaintiff has called an engineer and qualified him as plaintiff has called an engineer and qualified him as an expert in managing reservoirs. Which of the following opinions by the plaintiff's expert is the court most likely to admit?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}