{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1a1d1968-e719-5fc2-b29b-08c560e9258f","task_key":"all--test--1a1d1968-e719-5fc2-b29b-08c560e9258f","task_revision_id":"1","upstream_id":"","short_description":"A painter working for a construction company was working with a group of…","config":"all","split":"test","body":"{\"choices\":\"['Yes, because the painter is not qualified to make such statements regarding non-painting duties.', 'Yes, because the painter is not a participant as a negligent party in the catastrophic mistake that occurred.', 'No, because the painter works for the same company as the negligent workers, and he made his statements within the scope and course of his employment.', 'No, because the painter may testify as a subcontractor to what he observed while on the location.']\",\"question\":\"A painter working for a construction company was working with a group of construction co-workers on a house construction when a section of the house fell down on the owner. The painter declared, \\\"I knew they should have reinforced the standing wall better than they did! It is all their fault!\\\" It turned out that he was referring to co-workers employed by the same construction company. The owner filed suit for damages against the construction business. He tried to put the statement from the painter into evidence but the defendant objected. Will the court sustain the company's objection to the admission of the painter's statements?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}