{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"60143eec-a57b-55f6-af55-8b099cbd4d91","task_key":"bbeh~5fsportqa--full--60143eec-a57b-55f6-af55-8b099cbd4d91","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: In a basketball match, if a player from Team A last touches the ball before it goes out of bounds, but the player was pushed by a player from Team B which was unnoticed by the referees, which team should ideally get the possession of the ball, considering the official rules and the ethical aspects of the game?     A. Team A  B. Team B  C. The decision should be made by the referees  D. The decision should be made by the team captains   \\nSub-Question 1: According to the official basketball rules, which team gets the possession of the ball when the player who last touches the ball before it goes out of bounds was from Team A?     A. Team A  B. Team B  C. The team that did not touch the ball last  D. The team that touched the ball last   \\nSub-Question 2: If the referees do not notice a foul, what is the general course of action followed according to the official rules of basketball?     A. The game continues without acknowledging the foul  B. The foul is acknowledged after reviewing the game footage  C. The game is stopped until the foul is acknowledged  D. The foul is acknowledged based on the claims of the players\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}