{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"6ece9ff8-2d96-5e74-9345-2597d7b03517","task_key":"bbeh~5fsportqa--full--6ece9ff8-2d96-5e74-9345-2597d7b03517","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  What is the likely response of opponent Player B (known for his aggressive baseline play and powerful forehand) when Player A, who is known for his excellent drop shots, employs a strategy of frequent drop shots during their match on a clay court? Choices: A) Player B will consistently charge the net to counter the drop shots. B) Player B will stay on the baseline and attempt to power through the drop shots. C) Player B will start using more slice shots to counter the drop shots. D) Player B will try to serve and volley to avoid drop shots.\\nSub-Question 1: What is the typical response of an aggressive baseline player when faced with frequent drop shots? Choices: A) The player will stay on the baseline and try to power through the drop shots. B) The player will charge the net to counter the drop shots. C) The player will start using more slice shots. D) The player will try to serve and volley to avoid drop shots.\\nSub-Question 2: How does a clay court affect a player's approach to countering drop shots? Choices: A) Players are more likely to stay on the baseline due to the slower court speed. B) Players are more likely to charge the net due to the high bounce of the ball. C) Players are more likely to use slice shots due to the high bounce of the ball. D) Players are more likely to serve and volley due to the slower court speed.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}