{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"49630f43-c7b8-5f46-8c82-06272da34f3f","task_key":"default--test--1595","task_revision_id":"1","upstream_id":"1595","short_description":"A homeowner was injured when he slipped and fell in a puddle of water on his…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the manufacturer was aware that the installer used their logo.', 'Yes, because the installer was a franchisee of the manufacturer.', 'No, because the homeowner was responsible for choosing the installer.', \\\"Yes, because the installer's truck displayed the manufacturer's logo.\\\", 'No, because the manufacturer only supplied nondefective materials.', \\\"No, because the leaks were due to the installer's negligence, not the manufacturer's.\\\", 'No, because a franchisor has no duty to supervise the conduct of a franchisee.', 'Yes, under the rule of vicarious liability.', 'No, under the rule that a manufacturer is liable only for defects in a product that existed at the time the product left the hands of the manufacturer.', 'Yes, under the rule of apparent agency.']\",\"question\":\"A homeowner was injured when he slipped and fell in a puddle of water on his sunroom floor; the water had accumulated on the floor during a rainstorm because of leaks in the roof. The roof's manufacturer had supplied nondefective materials to the installer, who was a franchisee (and not an employee) of the manufacturer. The leaks resulted from the carelessness of the installer during the from the carelessness of the installer during the installation of the roof. The installer's truck, which had been parked in front of the homeowner's house during the roof installation, bore the manufacturer's logo. The manufacturer was aware that the truck and the literature supplied by the installer both displayed the manufacturer's logo. Is there any basis for a claim by the homeowner against the manufacturer?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}