{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"45548b47-d700-5905-a49c-d124795b8d06","task_key":"bbeh~5fsportqa--full--45548b47-d700-5905-a49c-d124795b8d06","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:     How does the sequence of actions unfold in a professional basketball game when a player gets seriously injured and there is a simultaneous equipment malfunction, considering the standard procedures of the NBA?     A. The game continues without interruption until the next timeout, after which the injured player is assisted and the equipment malfunction is addressed.  B. The game is immediately paused, the injured player is assisted, and then the equipment malfunction is addressed.  C. The game is immediately paused, the equipment malfunction is addressed, and then the injured player is assisted.  D. The game continues without interruption until halftime, after which the injured player is assisted and the equipment malfunction is addressed.   \\nSub-Question 1:     What is the standard procedure when a player gets seriously injured during an NBA game?     A. The game continues without interruption until the next timeout, after which the injured player is assisted.  B. The game is immediately paused and the injured player is assisted.  C. The game continues without interruption until halftime, after which the injured player is assisted.  D. The injured player is assisted during the next break in play, which could be a timeout or the end of a quarter.   \\nSub-Question 2:     What is the standard procedure when there is an equipment malfunction during an NBA game?     A. The game continues without interruption until the next timeout, after which the equipment malfunction is addressed.  B. The game is immediately paused and the equipment malfunction is addressed.  C. The game continues without interruption until halftime, after which the equipment malfunction is addressed.  D. The equipment malfunction is addressed during the next break in play, which could be a timeout or the end of a quarter.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}