{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0a3f7e36-0321-5e3d-b5a4-f6d18e2398c9","task_key":"bbeh~5fsportqa--full--0a3f7e36-0321-5e3d-b5a4-f6d18e2398c9","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why might Replay Officials be particularly challenging to implement effectively in American Football games?  A) The dynamic and continuous nature of the gameplay makes it difficult to determine the precise moment for review. B) The high number of subjective calls that are made by on-field referees cannot be easily adjudicated by video review. C) The rules of American Football are complex and frequently updated, which may lead to inconsistencies in video review. D) The presence of numerous players and rapid play developments can obscure clear viewing angles for video analysis.\\nSub-Question 1: Why is it difficult to determine the precise moment for review by Replay Officials in American Football?  A) The fast-paced nature of the game means that plays transition quickly without clear breaks. B) The exact moment of a foul or rule infraction is often ambiguous and requires real-time judgement. C) Stoppage time is not clearly defined as in other sports, leading to confusion about when to initiate a review. D) The ball is often hidden in scrums or tackled piles, making it difficult to pinpoint when an infraction occurs.\\nSub-Question 2: What makes subjective calls in American Football challenging for Replay Officials to adjudicate?  A) The determination of what constitutes a catch can vary greatly depending on the situation and angle. B) Penalties such as pass interference involve judgement calls that may not be conclusive even with video evidence. C) The intent behind a player's action is often crucial and cannot always be discerned through video alone. D) Some rules are based on the officials' discretion, which cannot be replicated by a video review system.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}