{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"130ff470-42f4-5396-ad6d-3a9206d4635b","task_key":"default--test--1475","task_revision_id":"1","upstream_id":"1475","short_description":"A purchasing agent for a women's clothing store negotiated a contract with…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['There was an enforceable contract at the original price term, because the purchasing agent acted in good faith.', 'There was a voidable contract, because the purchasing agent should have known of the error.', 'There was no valid contract formed, because there was no mutuality of assent.', 'There was no valid contract formed, because the price was not agreed upon by both parties.', 'There was a valid contract formed, because the dressmaking company was responsible for their own error.', \\\"The contract is invalid, because the purchasing agent took advantage of the dressmaking company's error.\\\", 'There was an enforceable contract at the revised price term, because the error was corrected before delivery.', 'There was an enforceable contract at the original price term, because the mistake was unilateral.', 'There was a voidable contract, because the dressmaking company made an error.', 'There was an enforceable contract at the original price term, because the mistake resulted from an error in computation, not in judgment.']\",\"question\":\"A purchasing agent for a women's clothing store negotiated a contract with dressmaking company to purchase a specified quantity of khaki garments at a price of $75,000. One week later, the purchasing agent received a telephone call from the vice president of the dressmaking company, who informed her that the dressmaking company's sales representative had made an error in calculating the contract price. As a result, the vice president said that unless the women's clothing store agreed to pay an additional $15,000, the garments would not be delivered. If the purchasing agent should have known that the dressmaking company's original price term of $75,000 was in error, but agreed to the contract anyway, which of the following is the most accurate statement?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}