{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ae665aa5-e3d0-5914-9e9a-4c282dcb918a","task_key":"bbeh~5fsportqa--full--ae665aa5-e3d0-5914-9e9a-4c282dcb918a","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How might a football player quickly adjust his strategy when the ball suddenly changes direction due to an unexpected strong wind gust during a competitive match?  A) He would continue with the initial strategy regardless of the sudden change. B) He would stop and let another teammate handle the situation. C) He would quickly analyze the new trajectory of the ball, anticipate its landing point, adjust his position and speed accordingly, and possibly modify his intended action (like passing or shooting). D) He would ask the referee to pause the game.\\nSub-Question 1: How does the player's understanding of the game's dynamics help him to quickly analyze the new trajectory of the ball?  A) It doesn't play a significant role. B) He uses his understanding of the game's dynamics to predict the ball's new path based on the direction and strength of the wind. C) He uses his understanding of the game's dynamics to decide whether to continue playing or not. D) He uses his understanding of the game's dynamics to argue with the referee about the wind interference.\\nSub-Question 2: How does the player's physical fitness impact his ability to adjust his position and speed according to the ball's new trajectory?  A) Physical fitness doesn't play a significant role. B) A player in better physical condition can react more quickly and move faster to the ball's new landing point. C) A player in better physical condition can argue more effectively with the referee about the wind interference. D) A player in better physical condition can kick the ball harder against the wind.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}