{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"261f8940-4fb4-5d01-9adb-086382047fd0","task_key":"bbeh~5fsportqa--full--261f8940-4fb4-5d01-9adb-086382047fd0","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  Which of the following best describes the most effective strategy for a football referee to avoid collisions or obstructions with players, the ball, or other referees, considering both the positioning principles in football refereeing and the dynamics of player movements?  A) Always stay at the center of the field B) Follow the ball closely at all times C) Position themselves such that they can observe the game action without interfering with player movements D) Stay close to the goalpost throughout the game\\nSub-Question 1:  Which of the following is a key principle in football refereeing related to referee's positioning on the field?  A) Referees should always stay in one fixed position  B) Referees should always be in close proximity to the ball C) Referees should position themselves to best anticipate play and avoid interfering with player movements D) Referees should always be near the goalpost to monitor goal attempts\\nSub-Question 2:  Considering the dynamics of player movements during a football match, which of the following strategies would help a referee avoid collisions or obstructions?  A) Always staying at the center of the field B) Constantly running after the ball C) Keeping a safe distance from players and anticipating the direction of play D) Always staying near the goalpost\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}