{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"c54fa83b-58b1-5592-9505-712f0b2302f0","task_key":"default--test--1048","task_revision_id":"1","upstream_id":"1048","short_description":"A law bookstore entered into a written contract to purchase from the publisher…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['unenforceable, because the original contract terms should prevail.', 'enforceable, if the publisher could prove the existence of a calculation error.', 'unenforceable, because it is violative of the statute of frauds.', \\\"unenforceable, because the error resulted from the publisher's computational error.\\\", 'unenforceable, because the increase in price was not agreed upon at the time the contract was formed.', 'enforceable, because the owner agreed to the new terms.', 'enforceable, because the publisher had the right to correct his mistake.', 'enforceable, under the principle of promissory estoppel.', 'enforceable, because it was not supported by any new consideration.']\",\"question\":\"A law bookstore entered into a written contract to purchase from the publisher 100 copies of the latest edition of a certain casebook for $10 per book. Three days after the contract was formed, but prior to delivery of the casebooks, the publisher called the owner of the law bookstore and informed him that, because of a calculation error, the price for the casebooks should have been $11 per book, and the shipment could not be delivered unless the owner promised to pay that amount. The owner reluctantly agreed. The owner's agreement to pay $11 per book is\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}