{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6e102315-ec6e-523b-8ecc-1c3eaf6e8d02","task_key":"bbeh~5fsportqa--full--6e102315-ec6e-523b-8ecc-1c3eaf6e8d02","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: During a critical fourth quarter play in an American football game, the quarterback throws a pass to a receiver near the sideline at the opposing team's 30-yard line. As the receiver attempts to secure the ball, a defender makes contact with him. The receiver manages to keep both feet in bounds but falls to the ground as he catches the ball. The on-field officials rule it a complete pass, but the opposing coach challenges the call. What factors will the replay official consider to confirm or overturn the call of a complete pass?  A) Did the receiver have control of the ball before stepping out of bounds? B) Did the receiver maintain control of the ball after making contact with the ground? C) Was the contact by the defender initiated before the ball arrived? D) Was the receiver able to perform a football move or maintain control of the ball for a reasonable amount of time?\\nSub-Question 1: What constitutes a player having control of the football during a catch?  A) The player has the ball firmly in their grasp. B) The ball touches the ground but the player maintains control. C) The player has the ball long enough to become a runner. D) The player maintains control of the ball after any subsequent contact with the ground.\\nSub-Question 2: What are the requirements for a legal catch if a player goes to the ground during the process of making the catch?  A) The player must maintain control of the ball while touching the ground. B) The player must have both feet or any other part of the body, other than the hands, in bounds. C) The player must maintain control of the ball after any subsequent contact with the ground. D) The player must have the ball long enough to perform an act common to the game.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}