{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"136160a8-5f5d-515a-b742-36d8e98c4b1a","task_key":"all--test--136160a8-5f5d-515a-b742-36d8e98c4b1a","task_revision_id":"1","upstream_id":"","short_description":"A husband and wife were about to leave on vacation for one month. They thought…","config":"all","split":"test","body":"{\"choices\":\"['recover against the husband and wife only.', 'recover against the carpeting company only.', 'recover against either the husband and wife or the carpeting company.', 'not recover, because he was not in privity of contract.']\",\"question\":\"A husband and wife were about to leave on vacation for one month. They thought it would be a great idea to have new carpeting installed in their home while they were away. They contacted a reputable carpeting company, which offered to do the job for $7,000. The husband and wife agreed, gave a set of house keys to the carpeting company, and left on vacation. A week later, the carpeting company discovered that they had overbooked their installation staff and could not finish all of the jobs for which they had contracted. The carpeting company contacted a workman whom they had used before, and told him about the job they were to perform for the husband and wife. The workman agreed to do the installation in exchange for the payment from the husband and wife. When the husband and wife returned from vacation, they were happy with the work and sent a check for $7,000 to the carpeting company, which deposited the check. If the workman now seeks to recover for services he performed, he will\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}