{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"1cb1de73-bb61-5ddb-b741-895fe8fe736d","task_key":"default--test--1471","task_revision_id":"1","upstream_id":"1471","short_description":"A builder wanted to have security systems installed in a series of homes he was…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['The cost of the alarms from the second supplier, because they were an unforeseen expense.', '$110,000 because of an unanticipated changeof circumstances after the parties had entered into their contract.', 'only in quantum meruit, because by demanding $110,000 the alarm company repudiated its contract with the builder.', 'only in quantum meruit, because of the doctrine of commercial frustration.', '$100,000, because that was the original agreed price, regardless of changes in circumstances.', '$90,000, because the alarm company had to pay an additional cost.', '$100,000, because that was the contract price.', 'Nothing, because the builder did not agree to the new price.', \\\"$100,000, plus damages for the builder's delay in payment.\\\", \\\"$110,000, because the supplier's inability to supply burglar alarms was unforeseen.\\\"]\",\"question\":\"A builder wanted to have security systems installed in a series of homes he was building. He contacted several companies and asked them to submit bids for the installation work. An alarm company decided to submit a bid and, in turn, requested bids from several wholesalers for the burglar alarms it planned to use if it was awarded the job. A supplier submitted a bid to the alarm company that the latter used in computing the bid that it was preparing for the builder. On September 1, the alarm company sent the builder its bid, in which it proposed to install the security systems for $100,000. On September 9, the supplier notified the alarm company that it would be unable to supply any burglar alarms to them. On September 11, the builder sent the following fax to the alarm company: \\\"I hereby accept your offer to install the security systems for $100,000. \\\"The alarm company had to pay another wholesaler $10,000 above the price quoted by the supplier for the burglar alarms. As a result, the alarm company advised the builder that the total price for the job would have to be increased to $10,000. The builder replied that he would hold the alarm company to the initially agreed price of $100,000. The alarm company installed the security systems, but the builder has not yet paid them anything. In an action by the alarm company against the builder for services rendered, the alarm company will probably be able to recover\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}