{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"07fe0bb9-ef33-517c-bf30-71628a55c449","task_key":"default--test--1954","task_revision_id":"1","upstream_id":"1954","short_description":"A watch manufacturer has an agreement with a watch wholesaler in which the…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, a doubling of the output contract in these circumstances meets a good faith standard and is reasonable.', 'No, because the expansion is a disproportionate increase in product that exceeds contractual estimates and cannot be handled by the buyer.', 'No, because the manufacturer should have consulted with the wholesaler before increasing production.', 'Yes, because the manufacturer has the right to increase production as it sees fit.', 'Yes, because doubling the size of the output contract is not unreasonable and is only 500 more the top estimated limit.', 'No, because the wholesaler has the right to reject any amount that exceeds the estimated number.', 'Yes, because the contract does not specify a maximum production limit.', 'No, because a change of more than 10% is considered to be unreasonable and the basis for a breach of contract.', 'Yes, because the agreement states the wholesaler will purchase all watches made each month.', 'No, because the wholesaler cannot be expected to buy more than the estimated amount agreed upon.']\",\"question\":\"A watch manufacturer has an agreement with a watch wholesaler in which the wholesaler agrees to purchase all of the watches made each month by the manufacturer, within an estimated number of 1000 to 1500 watches per month. Although the agreement worked well at 1,000 watches per month, the manufacturer expanded its plant capacity and produced 2500 watches per month. The wholesaler could no longer pay for that volume of watches. The manufacturer sued the wholesaler to compel it to buy its full output each month as required by the contract. will the court likely grant the manufacturer's request?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}