{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"48046d58-fc3f-5b17-aff9-86cf10d7b977","task_key":"all--test--48046d58-fc3f-5b17-aff9-86cf10d7b977","task_revision_id":"1","upstream_id":"","short_description":"A collector owned a rare 16th-century tapestry. The collector contracted in…","config":"all","split":"test","body":"{\"choices\":\"['$120,000, for conversion.', '$108,000, because the dealer is entitled to a reasonable sum not exceeding 10% on the resale.', '$100,000, which is the contract price.', '$20,000, which covers the difference between the contract price and the sale price.']\",\"question\":\"A collector owned a rare 16th-century tapestry. The collector contracted in writing to sell the tapestry to a tapestry dealer for $100,000. The contract stipulated that delivery would be \\\"F. O. B. at the dealer's shop,\\\" with payment to be made one week after tender. When the dealer received the tapestry, he noticed that it had been damaged in transit. The dealer immediately contacted the collector and notified him that he was rejecting the tapestry because it had ripped apart and was becoming unwoven. The collector told the dealer that he would get back to him with re-shipping instructions. The collector did not make any further contact with the dealer. After four weeks, the dealer then sold the tapestry to a buyer for $120,000. If the collector sues the dealer for damages, the collector should recover\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}