{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"850f3ae0-3cb2-53d7-aa44-1063331e866f","task_key":"all--test--850f3ae0-3cb2-53d7-aa44-1063331e866f","task_revision_id":"1","upstream_id":"","short_description":"On February 1, a homeowner entered into a contract with a famous landscape…","config":"all","split":"test","body":"{\"choices\":\"['relieve the architect of liability, because the contract was assignable.', \\\"relieve the architect of liability, because the gardener's work was the cause of the homeowner \\\\x80\\\\x98s dissatisfaction.\\\", 'not relieve the architect of liability, because a personal service contract of this nature is non- delegable.', 'not relieve the architect of liability, unless there was an express contractual provision against assignments.']\",\"question\":\"On February 1, a homeowner entered into a contract with a famous landscape architect, to landscape her five-acre estate. The homeowner entered into the contract with the architect because she was pleased with the work he had done on her neighbor's gardens. The architect promised to complete the job before May 30, the date of the homeowner's annual garden party. On March 1, the homeowner left for a month's vacation. Unknown to her, the architect, because of previous commitments, assigned the contract to a gardener. After the gardener had completed 60 percent of the work, the homeowner returned and was displeased with the landscaping already completed. The homeowner contacted the architect, who informed her of the assignment to the gardener and told her that he was no longer obligated under their original contract. Disgruntled, the homeowner fired the gardener. If the homeowner asserts a claim against the architect based on breach of contract, the fact that the architect assigned the contract to the gardener will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}