{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bc8f7052-06db-5042-9bf3-ab1b255fe64f","task_key":"default--test--1190","task_revision_id":"1","upstream_id":"1190","short_description":"A company contracted with a builder to construct a new corporate headquarters…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['No, because the price change of structural steel was not significant enough to warrant a contract modification.', 'No, because there was no consideration for the modification of the contract.', 'No, because the modification was never reduced to a writing signed by the party to be charged.', 'No, because the company did not sign a new contract reflecting the price change.', 'Yes, because the modification was fair and equitable in view of the unanticipated increase in the price of structural steel.', 'Yes, because the builder completed the project on time despite the unforeseen circumstances.', 'Yes, because the builder had no control over the price increase of structural steel.', 'Yes, because the companyÕs promise was supported by consideration.', 'Yes, because the company was aware of the price increase and verbally agreed to it.', 'No, because the builder should have anticipated potential price fluctuations in materials.']\",\"question\":\"A company contracted with a builder to construct a new corporate headquarters for a fixed price of $100 million. At the time of the contract, structural steel was widely available and was included in the contract as a $6 million item. Before work began on the project, tornado damage shut down the production facility of the biggest structural steel supplier in the country, and the price of structural steel increased by 20% as a result. The builder informed the company of the steel price increase, and the parties then orally agreed to increase the project price to $101 million. The builder proceeded with construction and delivered the project on time. The company paid the builder $100 million but refused to pay the additional $1 million. If the builder sues the company for $1 million, is the builder likely to prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}