{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c704b54-c175-5680-b3b6-2b0c9395bb67","task_key":"all--test--2c704b54-c175-5680-b3b6-2b0c9395bb67","task_revision_id":"1","upstream_id":"","short_description":"A motorcycle collector and his friend had known each other for years. The…","config":"all","split":"test","body":"{\"choices\":\"['No, because by not inquiring further about the meaning of the phrase \\\"oldest motorcycle,\\\" the friend assumed the risk of being mistaken.', 'No, because neither the friend nor the collector had reason to know of the other\\\\'s interpretation of the words \\\"oldest motorcycle.\\\"', \\\"Yes, so long as a reasonable person in the friend's position would have considered the letter as referring to the 1987 model.\\\", \\\"Yes, because the friend had no reason to know of the collector's meaning and the collector could have avoided the confusion by more careful drafting.\\\"]\",\"question\":\"A motorcycle collector and his friend had known each other for years. The collector sent a letter to the friend that said, \\\"I am willing to sell you my oldest motorcycle for $4,000.\\\" The friend sent the collector an email saying, \\\"I accept your offer.\\\" The collector owned three motorcycles: a 1985 model, a 1987 model, and a 1992 model. The friend did not know about the 1985 model, which was the one the collector intended to sell. The collector was unaware that the friend intended to buy the 1987 model. When the collector tendered the 1985 model, the friend refused to accept it. Is the friend likely to prevail in a breach of contract suit against the collector?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}