{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f8ec3296-04af-5444-a8ff-cc844a8e95a5","task_key":"bbeh~5fsportqa--full--f8ec3296-04af-5444-a8ff-cc844a8e95a5","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  How does the penalty shootout procedure in football unfold under various realistic scenarios? A) Each team takes turns to take three penalties each and the team with the most goals wins. B) If the scores are level after five penalties each, sudden death ensues where each team takes turns to take one penalty and the first team to miss while the other scores, loses. C) A coin flip determines which team goes first and the same player cannot take consecutive penalties. D) The goalkeeper who finishes the game must be the one to face the penalties.\\nSub-Question 1:  How does the referee manage the penalty shootout process? A) The referee decides which player shots the first ball. B) The referee tosses a coin to decide which team goes first. C) The referee ensures that only eligible players take part in the shootout. D) The referee ensures that the goalkeeper remains on the goal line until the ball is kicked.\\nSub-Question 2:  How are the players' actions regulated during a penalty shootout? A) The player taking the penalty must kick the ball forward. B) The player taking the penalty cannot touch the ball again until it has touched another player. C) The goalkeeper must remain on the goal line until the ball is kicked. D) The players of both teams remain in the center circle except the player taking the penalty and the two goalkeepers.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}