{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"299a7e64-3334-5699-b600-8839b4345f18","task_key":"bbeh~5fsportqa--full--299a7e64-3334-5699-b600-8839b4345f18","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: What outcome would likely be the result if, during an NFL playoff game, a Replay Officials review is initiated for a contested touchdown pass that was initially ruled complete on the field, but subsequent multi-angle replays reveal that the receiver only had one foot in bounds at the time of the catch and the opposing coach had thrown the challenge flag within the allowed time frame?  A) The touchdown is upheld. B) The touchdown is overturned and the pass is ruled incomplete. C) The play is ruled a fumble and the opposing team gains possession. D) The game clock is reset to the time of the contested catch.\\nSub-Question 1: Under NFL rules, how many feet must a receiver have in bounds for a pass to be completed?  A) One foot. B) Two feet. C) Three feet. D) It depends on the receiver's position.\\nSub-Question 2: In the NFL, within what time frame must a coach throw a challenge flag for a play to be eligible for a Replay Officials review?  A) Before the next play begins. B) Within two minutes of the play. C) Anytime before the two-minute warning. D) During the play in question.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}