{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"mmlu-pro","formal_name":"MMLU-Pro","introduction":"MMLU-Pro rebuilds MMLU with ten answer options instead of four and removes items that no longer separate models. Questions span fourteen academic and professional subjects.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"37c830c3-8e53-582d-87a1-31584a0d307e","task_key":"default--test--1438","task_revision_id":"1","upstream_id":"1438","short_description":"Two friends were members of a fraternity at their college. During homecoming…","config":"default","split":"test","body":"{\"category\":\"law\",\"options\":\"['prevail, because the first friend intended to place the second friend in apprehension.', 'prevail, because the first friend showed gross negligence.', 'not prevail, because there was no physical harm inflicted on the second friend.', \\\"prevail, because the first friend's actions were reckless and dangerous.\\\", 'prevail, because there was a substantial certainty that the second friend would be placed in apprehension.', 'not prevail, because the second friend was unaware that the gun was loaded.', 'not prevail, because the first friend did not actually intend to harm the second friend.', 'not prevail, because the second friend consented to participate in the Russian roulette game.', 'not prevail, because the second friend was under the influence of alcohol.']\",\"question\":\"Two friends were members of a fraternity at their college. During homecoming weekend, the fraternity held a big party, which was widely attended by students and alumni. At the party, the two friends were horsing around when the first friend pulled out a revolver. He gave the revolver's cylinder a spin, put the barrel of the gun to the second friend's head and, intending to scare the second friend, said, \\\"Hey, man, let's play Russian roulette. \\\" The second friend, who had consumed a fifth of bourbon, was very intoxicated. Thinking that the gun was unloaded, the second friend responded, \\\"Sure, dude. \\\" The first friend then pulled the trigger. The gun just made a clicking sound as the hammer fell on an empty chamber of the cylinder. The two friends both laughed and continued partying. Unbeknownst to the second friend, the first friend had placed bullets in two of the revolver's six firing chambers before making his challenge. The next day, the second friend found out that the gun was, in fact, loaded when the first friend pulled the trigger. If the second friend asserts a tort action against the first friend, he will most likely\",\"src\":\"ori_mmlu-professional_law\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}