{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"d8aadd52-2809-560f-beca-33abb8785464","task_key":"bbeh~5fsportqa--full--d8aadd52-2809-560f-beca-33abb8785464","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why might a player be penalized during a tennis match under the Continuous Play rule, even if the player has not exceeded the 25-second limit between points?  A) The player took a break to tie his shoelaces. B) The player paused to adjust his racquet strings. C) The player took more than 90 seconds on the changeover. D) The player stopped to greet a fan in the audience.\\nSub-Question 1: Why might a player need to take a break to tie his shoelaces during a match?  A) The player wants to delay the game. B) The player's shoelaces came untied. C) The player wants to rest. D) The player is trying to disrupt the opponent's rhythm.\\nSub-Question 2: Why might the 90-second limit on changeovers be important in maintaining continuous play?  A) To allow the players to rest. B) To allow the spectators to take a break. C) To keep the game moving at a steady pace. D) To allow the players to strategize for the next point.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}