{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"64c8dc90-d29b-5690-9dec-b248519674c9","task_key":"all--test--64c8dc90-d29b-5690-9dec-b248519674c9","task_revision_id":"1","upstream_id":"","short_description":"A buyer signed a contract to purchase a newly built home from the…","config":"all","split":"test","body":"{\"choices\":\"['No, the buyer had an obligation to make a full inspection and discover any defects prior to paying the full price and taking occupancy.', 'No, the law regarding a building purchase is based on the premise \\\"Let the buyer beware.\\\"', 'Yes, every builder-seller warrants that a newly constructed home is built in a good and workmanlike manner.', 'Yes, the seller has strict liability to the buyer for any defects that arise within one year of the sale.']\",\"question\":\"A buyer signed a contract to purchase a newly built home from the builder-seller. After paying the sale price and taking occupancy, the buyer experienced drainage and interior flooding problems with the home. The city issued citations against the owner for not following code regulations regarding construction on a flood-prone slope. The buyer asked the seller to repair all of the defects at his own expense but the seller refused. The buyer brought an action in court alleging breach of warranty against the builder-seller. Will the court hold the builderseller liable to live up to any warranties in favor of the buyer?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}