{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2987d67c-6524-5573-8684-2409a7dbbe43","task_key":"bbeh~5fsportqa--full--2987d67c-6524-5573-8684-2409a7dbbe43","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How does the implementation of Replay Officials in American Football affect the decision-making process during critical game situations where both a potential scoring play and a player safety concern occur simultaneously?  A) It allows for real-time decision-making without game interruption. B) It provides a post-play review system that prioritizes player safety over scoring validation. C) It creates a hierarchical review process where scoring plays are reviewed before player safety concerns. D) It integrates a simultaneous review protocol for both scoring plays and player safety concerns.\\nSub-Question 1: How do the Replay Officials determine the priority of review when a scoring play is challenged at the same time as a potential safety violation is identified?  A) The scoring play is automatically reviewed first. B) The potential safety violation is automatically reviewed first. C) A predetermined protocol defines the order of review based on the game clock. D) The referees decide the order of review on a case-by-case basis.\\nSub-Question 2: In the event of a simultaneous review of a scoring play and a safety concern, how do the Replay Officials ensure that the final decision does not overlook any critical aspect of the play?  A) It allows for separate officials to review each aspect independently. B) It mandates a single official to review both aspects sequentially. C) It requires a consensus among all officials after a joint review. D) It provides additional review time exclusively for player safety analysis.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}