{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"76ad6b76-c5a7-5a2f-bfab-46f8be8a54b1","task_key":"bbeh~5fsportqa--full--76ad6b76-c5a7-5a2f-bfab-46f8be8a54b1","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How might a volleyball team adjust their strategy to counter an opponent with a strong middle blocker and a weak outside hitter, considering the team's current strengths in serving and setting?  A) Increase the frequency of back row attacks B) Focus more on serving to the weak outside hitter C) Change the team's formation to a 6-2 setup D) Implement a quick set strategy to bypass the strong middle blocker \\nSub-Question 1: How can a volleyball team exploit an opponent's weak outside hitter?  A) By targeting serves to the weak outside hitter B) By avoiding the weak outside hitter during attacks C) By setting up a strong block against the weak outside hitter D) By forcing the weak outside hitter to receive more often\\nSub-Question 2: How can a volleyball team utilize a strong serving and setting strength to bypass a strong middle blocker?  A) By implementing a quick set strategy B) By increasing the frequency of back row attacks C) By changing the team's formation to a 6-2 setup D) By focusing more on serving to the weak outside hitter\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}