{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"2f1ff7e5-8a3a-5d12-a71e-310c67202684","task_key":"bbeh~5fsportqa--full--2f1ff7e5-8a3a-5d12-a71e-310c67202684","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question:   When analyzing a basketball game, what combination of techniques can help quickly and accurately determine players' positions and movements on the court? A) Utilizing real-time player tracking technology B) Analyzing game footage manually C) Using statistical data from previous games D) Depending on crowd reactions and commentary   \\nSub-Question 1:   Which factors should be considered when using real-time player tracking technology to analyze players' positions and actions? A) Player's speed and direction B) Player's distance from the hoop C) Player's interactions with other players D) The current score of the game   \\nSub-Question 2:   What aspects should be focused on when manually analyzing game footage to determine players' positions and actions? A) Player's body language and gestures B) The trajectory of the ball C) The positions of the referees D) The tactics employed by the team\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}