{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5802233e-edff-5512-9d3e-8c8512f86a74","task_key":"all--test--5802233e-edff-5512-9d3e-8c8512f86a74","task_revision_id":"1","upstream_id":"","short_description":"On January 1, a seller entered into an agreement of sale to convey his farm to a…","config":"all","split":"test","body":"{\"choices\":\"['the buyer held the legal title to the farm.', 'the seller held the legal title to the farm.', 'the buyer held the equitable title to the farm.', 'the seller held the equitable title to the farm.']\",\"question\":\"On January 1, a seller entered into an agreement of sale to convey his farm to a buyer. According to the contract, settlement (or closing) was to take place on March 1 of the same year. On January 20, a barn on the property was destroyed by fire. When the buyer was notified of the loss the following day, he contacted the seller and told him that the deal was off because of the fire. As a result, the buyer filed suit against the seller on January 25 for a refund of his down payment of $5,000. In his suit, the buyer will be unsuccessful and will be obliged to complete the contract if the court determines that on January 20\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}