{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9f7ce820-532b-51ad-8547-af9c7afc56e5","task_key":"default--test--1424","task_revision_id":"1","upstream_id":"1424","short_description":"A producer of widgets contacted a manufacturer of boxes about the possibility of…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['unenforceable, because their written contract on the same date was the final integration of the terms of their agreement.', 'unenforceable, because the manufacturer was under a pre-existing duty.', 'unenforceable, due to the Statute of Frauds.', \\\"enforceable, as the parties' intentions were clear and unambiguous.\\\", \\\"unenforceable, because there was inadequate consideration to support the manufacturer's unilateral promise.\\\", 'enforceable, due to the oral agreement being a modification of the written contract.', 'enforceable, because the manufacturer was motivated by a business advantage.', 'enforceable.', 'unenforceable, as the agreement to supply more boxes was not in writing.', 'unenforceable, because the agreement was violative of the parol evidence rule.']\",\"question\":\"A producer of widgets contacted a manufacturer of boxes about the possibility of purcIiasing specially designed boxes for shipping the widgets to customers. The producer sent the manufacturer a sheet containing the dimensions of the boxes' needs, as well as the logo to be placed on each box. After negotiations, the parties entered into a written contract whereby the manufacturer agreed to supply the producer with as many of the boxes as the producer may require up to a maximum of 10,000 per month. Later that day, the producer telephoned the manufacturer and said that they were running a one-time promotion and expected to sell an unusually large number of widgets. Therefore, for the first month only, they would need 15,000 boxes. The manufacturer agreed to this request. The manufacturer's oral agreement to supply the producer with the 15,000 boxes for the first month would most likely be held\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}