{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8d470131-6854-52d5-a0e1-08791997eb57","task_key":"bbeh~5fsportqa--full--8d470131-6854-52d5-a0e1-08791997eb57","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  How did Jennifer's team win the volleyball match despite being behind in the first two sets?  A) They won by default as the opposing team got disqualified. B) They changed their team strategy and made a comeback in the next three sets. C) The match was cancelled due to bad weather. D) Jennifer's team lost the match.\\nSub-Question 1:  What was the strategy change that Jennifer's team implemented in the third set?  A) They started using a 4-2 formation instead of a 6-2. B) They substituted Jennifer with a more experienced player. C) They started serving from the left side of the court instead of the right. D) They started focusing more on offensive plays than defensive.\\nSub-Question 2:  How did this strategy change affect the performance of Jennifer's team in the final three sets?  A) It allowed them to score more points through successful attacks. B) It weakened their defense, leading to the opposing team scoring more points. C) It didn't have any significant impact on their performance. D) It led to a decrease in the team's morale.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}