{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9cf3a0d2-48e2-5ec4-a7a6-fa87cc9390db","task_key":"bbeh~5fsportqa--full--9cf3a0d2-48e2-5ec4-a7a6-fa87cc9390db","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How does a tennis Roving Umpire ensure fair play during a match, particularly concerning player conduct and rule enforcement?  A) By observing player behavior and issuing warnings for unsportsmanlike conduct. B) By strictly enforcing the rules of the game, including the foot fault rule. C) By intervening in the case of disputed line calls. D) By ensuring the court is in good condition before the match begins.\\nSub-Question 1: How does a Roving Umpire identify unsportsmanlike conduct?  A) By observing for any verbal abuse or physical violence towards another player. B) By monitoring for any attempts by a player to distract their opponent during a point. C) By checking if a player is intentionally wasting time between points. D) By watching for any unnecessary noise or movement that could disrupt the opponent.\\nSub-Question 2: How does a Roving Umpire enforce the foot fault rule?  A) By carefully watching the server's feet during the service motion. B) By issuing a fault call if the server's foot crosses the baseline before the ball is hit. C) By warning the player about the possibility of a foot fault before the match begins. D) By penalizing the player if repeated foot faults are observed.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}