{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"642e898d-e43e-5f22-9fb6-83333ed741a7","task_key":"all--test--642e898d-e43e-5f22-9fb6-83333ed741a7","task_revision_id":"1","upstream_id":"","short_description":"For the last 20 years, a husband and wife had taken a vacation during the first…","config":"all","split":"test","body":"{\"choices\":\"['prevail, because the hotel knew that for the past 20 years the husband and wife always stayed in the most luxurious suite.', 'prevail, because the substitute accommodations offered by the hotel were not comparable to the suite they reserved.', 'not prevail, because the hotel did offer substitute accommodations at a $5,000 savings.', 'not prevail, because the husband and wife sustained no legal damages in renting a comparable suite at another hotel for the same rental.']\",\"question\":\"For the last 20 years, a husband and wife had taken a vacation during the first week in July. They would fly from their home to the city and spend the week at a hotel in the most luxurious suite. On March 1, the husband mailed a letter to the hotel, reserving the suite for the first week in July. The hotel then sent back a confirmation notice stating that it was reserving the suite for the husband and wife for the first week in July \\\"for the weekly rental of $25,000. \\\"When the husband and wife arrived at the hotel on July 1, the hotel, without legal excuse, informed the man and his wife that the hotel had rented the suite to another couple for the first week in July. Quite apologetic, the manager of the hotel offered the husband and wife the hotel's next best accommodation at a weekly rental of $20,000. The manager informed the husband and wife that the other suite was beautifully furnished, \\\"but not quite as luxurious as the suite they reserved. \\\" Visibly upset, the husband and wife rejected the manager's offer and relocated to another hotel, where they rented the other hotel's most luxurious suite for the first week in July at a cost of $25,000. If the husband and wife now sue the first hotel for breach of contract, they will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}