{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b85c8431-3eb6-589e-bb5c-3a1cfa21f230","task_key":"all--test--b85c8431-3eb6-589e-bb5c-3a1cfa21f230","task_revision_id":"1","upstream_id":"","short_description":"A manufacturer of widgets entered into a written agreement to deliver 500…","config":"all","split":"test","body":"{\"choices\":\"['not recover, because under the circumstances the manufacturer substantially performed.', 'not recover, because the buyer accepted delivery of the 490 widgets.', 'recover, because the manufacturer failed to perform his contractual obligation.', \\\"recover, because the manufacturer's failure to deliver the additional widgets resulted in a material breach.\\\"]\",\"question\":\"A manufacturer of widgets entered into a written agreement to deliver 500 widgets to a buyer. The contract provided that the widgets would be shipped C. O. D. The manufacturer subsequently delivered 490 widgets, which were accepted and paid for by the buyer. If the buyer brings suit for breach of contract against the manufacturer, the buyer will most likely\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}