{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"54c1dabd-56ff-59a5-91bf-f9d58e78dc25","task_key":"default--test--1898","task_revision_id":"1","upstream_id":"1898","short_description":"An avid stamp collector had amassed a large and valuable collection. Many of his…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"The price term is $78,000, and it is unenforceable due to the secretary's mistake.\\\", \\\"The price term is $75,000, and the cousin must pay an additional $3,000 due to the secretary's error.\\\", 'The price term is $78,000, and it is enforceable.', 'The price term is $75,000, but either party can rescind the contract, because there was a mutual mistake as to a basic assumption of fact.', 'The price term is $75,000, but the court may adjust the price to $78,000 if the cousin knew of the mistake.', 'The price term is $75,000, but the court will reform the price to $78,000 in order to correct a mistake in integration.', 'The price term is $75,000, and it is enforceable.', \\\"The price term is $75,000, but the collector can demand the correct price of $78,000 due to the secretary's error.\\\", 'The price term is $78,000, but either party can rescind the contract, because the collector signed the contract without noticing the mistaken price.', \\\"The price term is $78,000, but the court will reform the price to $75,000, because of the secretary's mistake.\\\"]\",\"question\":\"An avid stamp collector had amassed a large and valuable collection. Many of his stamps were extremely rare and highly coveted. On numerous occasions the collector had rejected offers to sell his prized collection. Finally, on December 1, the collector sent his cousin, who was also a stamp collector, the following letter:\\\"I've decided to part with my stamp collection. Since I want to keep it in the family, I'll sell you my entire collection for $75,000. \\\"The collector had dictated this letter to his secretary, who mistakenly typed \\\"$75,000,\\\" instead of $78,000 that the collector had specified. After typing the letter, the secretary gave it to the collector, who hastily signed it without noticing the mistaken price. The cousin received this letter on December 3. On December 5, the cousin sent a letter accepting the collector's offer. Which of the following correctly states the agreement's price term and its legal effect?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}