{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"91fbacc7-ac10-5165-adeb-09dd0f874dea","task_key":"all--test--91fbacc7-ac10-5165-adeb-09dd0f874dea","task_revision_id":"1","upstream_id":"","short_description":"A homeowner entered into a contract with an insurance company to purchase a fire…","config":"all","split":"test","body":"{\"choices\":\"['although she failed to make the last two premium payments, there was a bargained-for exchange.', \\\"the insurance company's duty to pay was not expressly conditioned on the homeowner's duty to make the payments.\\\", 'the insurance company was under an independent duty to pay for the loss.', 'the homeowner did not receive notice of cancellation.']\",\"question\":\"A homeowner entered into a contract with an insurance company to purchase a fire insurance policy for her new home. The policy provided that the insurance company promised to pay up to $500,000 if the said house was destroyed by fire or fire-related explosion, while the homeowner promised to pay a quarterly premium of $400. The homeowner failed to make the last two quarterly payments before her house burned to the ground. The insurance company refused to pay the homeowner for the loss because of her failure to make the last premium payments. In an action by the homeowner against the insurance company to recover for the loss of her house, the homeowner's best theory of recovery is that\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}