{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"4f99f9b0-c4d0-51f2-adc4-bb44011f86c0","task_key":"bbeh~5fsportqa--full--4f99f9b0-c4d0-51f2-adc4-bb44011f86c0","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   Why might a basketball game be interrupted or suspended due to equipment malfunction or player injuries?    A) A player sustains a serious injury and cannot be moved until medical assistance arrives. B) The basketball hoop breaks and there is no spare available. C) A player suffers a minor injury and insists on continuing the game. D) The scoreboard malfunctions and the officials cannot keep track of the game time and score.   \\nSub-Question 1:   Why might a player's injury lead to the suspension of a basketball game?    A) The player trips and falls but is able to continue playing after a short break. B) A player sustains a concussion and needs immediate medical attention. C) A player gets a minor scratch but decides to continue playing. D) The player suffers from cramps but recovers after a short rest.   \\nSub-Question 2:   Why might an equipment malfunction lead to the interruption of a basketball game?    A) The game ball deflates but there are spare balls available. B) The basketball hoop breaks and it takes time to replace it. C) The court gets wet due to a leak in the roof but can be quickly dried. D) The scoreboard malfunctions and there is no backup scoreboard available.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}