{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6bd3bee9-2f2b-5fb6-a9e2-c63cf969ac37","task_key":"default--test--1150","task_revision_id":"1","upstream_id":"1150","short_description":"A lumber mill contracted to deliver one thousand8-foot sheets of plywood to a…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Judgment for the store, because the mill failed to have a contingency plan in place.', 'Judgment for the mill, because their performance would be discharged by impossibility.', 'Judgment for the store, because the mill should have foreseen such a contingency occurring.', \\\"Judgment for the store, because the mill's duties of performance would not be excused.\\\", 'Judgment for the mill, because the forest fire caused an unforeseeable hindrance in performance.', 'Judgment for the mill, because their performance would be discharged by frustration of purpose.', 'Judgment for the mill, because the forest fire was an act of God and beyond their control.', 'Judgment for the store, because the mill had a contractual obligation to deliver the goods.', 'Judgment for the mill, because they had informed the store about the issue in advance.', 'Judgment for the store, because the mill could have sourced lumber from another location.']\",\"question\":\"A lumber mill contracted to deliver one thousand8-foot sheets of plywood to a home improvement store on the first of each month for 10 consecutive months starting June 1. The June, July, and August shipments were delivered on time and duly paid for. Then, on August 16, the lumber mill informed the store that the mill would not be able to meet the September 1 delivery date, because its lumber reserve had been destroyed by a forest fire. The mill then asked the store to excuse the mill from further performance. The store refused and demanded that the remaining shipments be delivered on time. When the September shipment failed to arrive, the store immediately brought suit for breach of contract. How would the court hold?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}