{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0109c58a-dc4f-588b-b838-5d46bd19bf4e","task_key":"bbeh~5fsportqa--full--0109c58a-dc4f-588b-b838-5d46bd19bf4e","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: During a contentious American Football playoff game, a pass is caught close to the sideline at a crucial moment in the fourth quarter. The play is automatically reviewed by the Replay Officials to determine both possession and if the receiver had both feet inbounds when the catch was made. Which of the following statements are correct regarding the use of the Replay Officials in this scenario?  A) Replay Officials can be used to review if the receiver had control of the ball before stepping out of bounds. B) Replay Officials are utilized to determine if the pass was caught within the time limits of the play clock. C) Replay Officials can be used to review if the receiver had both feet inbounds upon making the catch. D) Replay Officials are used to decide if any penalties should be applied for actions during the catch.\\nSub-Question 1: Prior to the Replay Officials review, the officials on the field ruled the pass complete. Which aspects would be evaluated by the Replay Officials to uphold or overturn the call?  A) The number of steps the receiver took before going out of bounds. B) Whether the receiver maintained control of the ball throughout the entire process of the catch. C) The position of the receiver's feet in relation to the sideline at the time of the catch. D) If the quarterback had crossed the line of scrimmage before the pass, which is not reviewable by the Replay Officials.\\nSub-Question 2: If the Replay Officials review results in the pass being ruled incomplete, which of the following could be valid reasons based on the video evidence?  A) The receiver only had one foot inbounds with possession of the ball. B) The receiver had both feet inbounds but did not maintain control of the ball when hitting the ground. C) The play clock had expired before the pass was thrown, although this is not a reviewable aspect by the Replay Officials. D) The receiver stepped out of bounds before establishing possession of the ball.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}