{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8c1fea11-1865-5b38-90b5-7d6c192d398d","task_key":"all--test--8c1fea11-1865-5b38-90b5-7d6c192d398d","task_revision_id":"1","upstream_id":"","short_description":"After a severe thunderstorm had struck the area, a homeowner discovered that her…","config":"all","split":"test","body":"{\"choices\":\"['The performance of the roofer would be a constructive condition precedent to the performance by the homeowner.', 'The performance by the homeowner would be an implied condition subsequent to the performance by the roofer.', 'The performances of the homeowner and the roofer would be constructive concurrent conditions.', 'The contract would be unenforceable, because it was oral.']\",\"question\":\"After a severe thunderstorm had struck the area, a homeowner discovered that her roof had suffered extensive damage. She telephoned a roofer to inquire about having the necessary repairs done to her roof. The roofer agreed to finish the work within 30 days, and the homeowner promised to pay$10,000. Which of the following statements is most accurate?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}