{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7a807bfb-199e-53ea-9b30-9c9b720e11c4","task_key":"all--test--7a807bfb-199e-53ea-9b30-9c9b720e11c4","task_revision_id":"1","upstream_id":"","short_description":"A paper manufacturing company contracted with a construction company for the…","config":"all","split":"test","body":"{\"choices\":\"['Yes, a company that performs construction work can always collect for what it has done on a job.', 'No, the construction company cannot change the specifications and hope to get paid.', 'No, the construction company is guilty of fraud and misrepresentation.', 'Yes, the construction company has substantially performed the contract.']\",\"question\":\"A paper manufacturing company contracted with a construction company for the building of a new warehouse. The contract provided that the construction company would be paid upon completion of the warehouse. The contractor finished the warehouse but it had some minor deviations from the plans due to substitution for certain building materials that were not available in the market. The construction company finished working and demanded payment. The paper company refused to pay based on deviations from the specifications. Will the construction company likely prevail in a breach of contract action against the paper company?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}