{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"7c47559f-5880-5005-bedd-4763631a37e1","task_key":"bbeh~5fsportqa--full--7c47559f-5880-5005-bedd-4763631a37e1","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:  During a professional tennis match, what factors could lead the umpire to make a ball call that might be contested by a player?  A) The ball bounced twice before the player could return it. B) The ball touched the net but still landed in the opponent's court. C) The ball landed on the line marking the boundary of the court. D) The ball was hit outside the court but was blown back in by the wind.\\nSub-Question 1:  What are the possible reasons a player might contest a ball call made by the umpire? A) The player believes the ball was 'in' while the umpire called it 'out'. B) The player’s coach was not ready when the ball was served. C) The player felt the umpire was favoring the opponent. D) The player thinks that the umpire's view was obstructed.\\nSub-Question 2:  In a professional tennis match, what are the tools or technologies that can be used to verify a contested ball call? A) Instant replay. B) Player's personal judgment. C) Hawk-Eye technology. D) Spectator's opinion.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}