{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu","formal_name":"MMLU","introduction":"MMLU asks four-way multiple-choice questions across 57 subjects, from elementary material to professional-level examinations. Since 2020 it has been the most widely cited single reference point for general capability.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/hendrycks/test","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"719a523a-e084-5881-8727-11fa61bbd5f3","task_key":"all--test--719a523a-e084-5881-8727-11fa61bbd5f3","task_revision_id":"1","upstream_id":"","short_description":"On September 1, a buyer contracted to purchase10,000 widgets from a seller for…","config":"all","split":"test","body":"{\"choices\":\"['The market price on September 1.', 'The market price on September 15.', 'The market price on September 20.', 'The market price on September 30.']\",\"question\":\"On September 1, a buyer contracted to purchase10,000 widgets from a seller for $1 per widget, delivery to be made no later than September 30. On September 15, a worldwide shortage of widgets caused a steep increase in the market price for widgets, so the seller decided not to deliver the widgets to the buyer. The seller sent a letter to the buyer, stating that the widgets would not be delivered. The buyer received the seller's letter on September 20. On October 15, the buyer filed suit against the seller for breach of contract. In determining the damages to which the buyer is entitled, which of the following would be taken as the market price of widgets?\",\"subject\":\"professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/hendrycks/test","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}