{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e2246e58-7eee-594f-86ef-8cc548682164","task_key":"bbeh~5fsportqa--full--e2246e58-7eee-594f-86ef-8cc548682164","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: What tactic would a football team likely employ on a 4th and long situation within their own territory, considering the team has a strong defense known for preventing opponents' scoring drives but a struggling quarterback who has had multiple turnovers in the game so far?  A) Attempt a pass play to gain the necessary yards for a first down B) Run the ball to secure a few yards and avoid a turnover C) Punt the ball to the opposing team to leverage their defensive strength D) Attempt a trick play to surprise the defense\\nSub-Question 1: What is the primary objective for a football team when faced with a 4th and long situation within their own territory?  A) To score a touchdown B) To gain enough yards for a first down C) To avoid giving the opposing team good field position D) To run down the clock\\nSub-Question 2: Given the quarterback's struggles with turnovers during the game, what would be the safest strategy for the team to maintain possession without increasing the risk of the opposing team scoring?  A) Aggressive passing plays B) Conservative run plays C) Special teams play D) Time-out to strategize\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}