{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"aa21fb30-c625-5660-986a-a337d259ee90","task_key":"all--test--aa21fb30-c625-5660-986a-a337d259ee90","task_revision_id":"1","upstream_id":"","short_description":"A homeowner listed his home for sale with a real estate broker. The written…","config":"all","split":"test","body":"{\"choices\":\"['No, because the broker engaged in no negotiations with the cousin.', 'No, because the closing occurred after the listing period had expired.', 'Yes, but only reimbursement for her expenses.', \\\"Yes, the full commission, because the homeowner accepted the cousin's offer to purchase during the listing period.\\\"]\",\"question\":\"A homeowner listed his home for sale with a real estate broker. The written six-month exclusive- right-to-sell listing agreement provided for the payment of a commission if the home sold. In accordance with the listing agreement, the broker promptly took reasonable steps to market the home, incurring expenses for her efforts. Five months into the listing period, without involving the broker, the homeowner accepted an offer to purchase from his cousin. The broker learned of the contract only when the sale of the home to the cousin closed, one month after the listing period had expired. Is the broker entitled to any payment?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}