{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2c5f87f4-8b22-59e4-ab67-785304569273","task_key":"default--test--1467","task_revision_id":"1","upstream_id":"1467","short_description":"A motorcycle collector and his friend had known each other for years. The…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"[\\\"No, because it was the friend's responsibility to know the details about the collector's motorcycles before agreeing to purchase.\\\", 'No, because by not inquiring further about the meaning of the phrase \\\"oldest motorcycle,\\\" the friend assumed the risk of being mistaken.', 'No, the friend won\\\\'t prevail as he misunderstood the term \\\"oldest motorcycle\\\".', \\\"Yes, so long as a reasonable person in the friend's position would have considered the letter as referring to the 1987 model.\\\", 'No, because neither the friend nor the collector had reason to know of the other\\\\'s interpretation of the words \\\"oldest motorcycle.\\\"', 'Yes, the friend will prevail as the collector falsely advertised the motorcycle model.', 'No, because the friend should have asked for clarification about which model was being sold before agreeing to the purchase.', 'Yes, because the friend could have reasonably assumed that the \\\"oldest motorcycle\\\" referred to the 1987 model, given his knowledge.', \\\"Yes, because the friend had no reason to know of the collector's meaning and the collector could have avoided the confusion by more careful drafting.\\\", 'Yes, because the collector should have clarified which model he was referring to in the letter.']\",\"question\":\"A motorcycle collector and his friend had known each other for years. The collector sent a letter to the friend that said, \\\"I am willing to sell you my oldest motorcycle for $4,000.\\\" The friend sent the collector an email saying, \\\"I accept your offer.\\\" The collector owned three motorcycles: a 1985 model, a 1987 model, and a 1992 model. The friend did not know about the 1985 model, which was the one the collector intended to sell. The collector was unaware that the friend intended to buy the 1987 model. When the collector tendered the 1985 model, the friend refused to accept it. Is the friend likely to prevail in a breach of contract suit against the collector?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}