{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"5ac85fbc-1507-591b-bd4e-5005bc9d6ab0","task_key":"default--test--1482","task_revision_id":"1","upstream_id":"1482","short_description":"Each year, the local boys' and girls' club sponsors a \"Baseball Card Bonanza. \"…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['Yes, because the boy was the true owner of the card and was entitled to the benefit of the bargain.', \\\"Yes, because the collector used his knowledge of the card's value to exploit the boy.\\\", 'Yes, because the collector did not disclose the true value of the card.', \\\"Yes, because the collector took advantage of the boy's lack of knowledge about the card's value.\\\", 'No, because the collector made no false representations of fact.', 'No, because the boy willingly accepted the offer made by the collector.', 'Yes, because the collector was obligated to disclose the true value of the card.', \\\"No, because the boy was not justified in relying on the collector's offer.\\\", \\\"No, because the boy had the opportunity to decline the collector's offer.\\\", \\\"No, because the collector didn't force the boy to sell the card.\\\"]\",\"question\":\"Each year, the local boys' and girls' club sponsors a \\\"Baseball Card Bonanza. \\\" During the three- day event, collectors of baseball cards gather, exchanging and swapping old cards. A boy and a girl, both 12-year-old youngsters at the bonanza, were looking at each other's cards when they were approached by a 32-year-old collector. The collector asked the boy and girl if he could look through their baseball cards. The boy and girl nodded affirmatively. As the collector was flipping through the boy's cards, he found a 1948 card featuring Branch Rickey, the former owner of the Brooklyn Dodgers. The collector, who knew the card was worth more than $500, offered the boy 50 cents for the card. The boy thought the card had very little value because he knew that Rickey was not a major league baseball player. The boy accepted the 50 cents and gave the Rickey baseball card to the collector. The next day the boy found out that the baseball card was worth$600. If the boy asserts a claim against the collector for deceit, will the boy prevail?\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}