{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"31fc3e25-00c5-58f6-b4bd-d86a4462f493","task_key":"bbeh~5fsportqa--full--31fc3e25-00c5-58f6-b4bd-d86a4462f493","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why would a tennis player decide to use a challenge during a match? A) To argue with the umpire B) To question the accuracy of a line call C) To take a short break D) To distract the opponent\\nSub-Question 1: What is a line call in tennis? A) When a player calls for a timeout B) When a player hits the ball outside the court C) When an official determines whether a shot landed in or out D) When a player challenges the opponent's serve\\nSub-Question 2: What happens when a player uses a challenge? A) The player loses a point B) The umpire reviews the line call using video technology C) The player gets to replay the point D) The opponent gets to serve again\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}