{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b8ca3827-3478-5709-bef5-fb9d6b3a9358","task_key":"all--test--b8ca3827-3478-5709-bef5-fb9d6b3a9358","task_revision_id":"1","upstream_id":"","short_description":"A homeowner wished to have his house painted. He contacted a number of house…","config":"all","split":"test","body":"{\"choices\":\"['that the homeowner should have been aware of the mistaken transmission, because of the disparity between its bid and the others.', 'that the telegraph company should be liable as an independent contractor.', 'that the homeowner was under an affirmative duty to investigate all submitted bids.', 'that the mistake made the contract unconscionable.']\",\"question\":\"A homeowner wished to have his house painted. He contacted a number of house painters in his area and asked them to submit bids to do the work. The homeowner received 10 bids. The first nine offered to paint the house for amounts ranging from $10,000 to $12,500. The 10th bid was sent by telegram and was supposed to be for $10,000. However, the telegraph company made a mistake and transmitted the bid as $1,000. The homeowner immediately accepted the 1 0th bid, but the 1 0 painter refused to perform. The I 0th painter's best defense in an action for breach of contract by the homeowner would be\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}