{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4b4c6c5a-c6b2-5b20-be7e-a9914df415ae","task_key":"all--test--4b4c6c5a-c6b2-5b20-be7e-a9914df415ae","task_revision_id":"1","upstream_id":"","short_description":"A homeowner said to a roofer, \"My roof leaks. I think the old tiles are cracked.…","config":"all","split":"test","body":"{\"choices\":\"['A bilateral contract was formed when the roofer purchased the materials and equipment needed to do the job.', 'A bilateral contract was formed when the roofer said, \\\"Sure, if I can clear my busy schedule. \\\"', 'The homeowner made an offer that proposed a unilateral contract, and the offer became irrevocable when the roofer purchased the materials and equipment needed for the job.', \\\"The homeowner made an offer that proposed a unilateral contract, and the roofer manifested an intent to accept the offer when he began performance by unloading the materials and equipment at the homeowner's house.\\\"]\",\"question\":\"A homeowner said to a roofer, \\\"My roof leaks. I think the old tiles are cracked. If you will replace them with all new tiles, I will pay you $5,000. \\\" The roofer replied, \\\"Sure, if I can clear my busy schedule. \\\" The homeowner then remarked, \\\"That's all right, but let me know soon. \\\" Three days later, the roofer drove his pickup truck to the homeowner's home and unloaded the materials and equipment needed to perform the roofing job. When the homeowner looked out his window and saw what was transpiring, he immediately ran outside and exclaimed, \\\"Stop! The deal's off. I decided to repair the roof myself. \\\"In an action by the roofer against the homeowner for breach of contract, which of the following would provide the roofer with his best theory of recovery?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}