{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"62b4327d-c1f7-5efb-b845-b3c879433213","task_key":"default--test--1425","task_revision_id":"1","upstream_id":"1425","short_description":"A construction contractor brought a breach of contract claim in federal court…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because the appellate court must give due regard to the trial judge's opportunity to determine witness credibility.\\\", 'Yes, because the contractor and homeowner had different views on the scope of the work.', 'Yes, because a judge must set forth findings of fact in a written opinion or memorandum of decision.', \\\"No, because the judge's findings of fact were stated on the record.\\\", 'Yes, because the homeowner was found in favor.', 'No, because the contractor brought the claim in federal court.', 'Yes, because there were disputed issues of fact at trial.', \\\"No, because the contractor didn't provide sufficient evidence to support the claim.\\\", 'Yes, because the appellate court has the power to overturn the findings.', 'No, because the contractor failed to object to the findings when the judge stated them in open court.']\",\"question\":\"A construction contractor brought a breach of contract claim in federal court against a homeowner who had hired the contractor to build an apartment over an existing garage. The action turned on the scope of the work covered by the contract. The contractor and the homeowner were the only witnesses at the bench trial, and they strongly disagreed about the scope of the work. At the end of the trial, the judge stated findings of fact on the record but never issued a written opinion. Neither party objected to the findings. The judge found in favor of the homeowner, and the contractor appealed. Is the appellate court likely to overturn the findings?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}