{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"e4fed28d-bf83-5ed6-aafe-ad6386d1e2f3","task_key":"bbeh~5fsportqa--full--e4fed28d-bf83-5ed6-aafe-ad6386d1e2f3","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Which player legally secured possession of the football before stepping out of bounds, when considering the following sequence of events during an American Football game: Player A caught the ball with one foot inbounds and the other foot airborne, and before the second foot touched the ground out of bounds, Player B, who was inbounds, grabbed the ball from Player A's hands?  A) Player A B) Player B C) Neither, it was an incomplete pass D) The play is under review for further determination\\nSub-Question 1: According to NFL rules, what constitutes a legal catch when a player is near the boundary lines?  A) Control of the ball with two feet or any part of the body other than the hands inbounds B) Maintaining control of the ball while going to the ground in the act of catching C) Both A and B are required for a legal catch D) Simply touching the ball while any part of the body is inbounds\\nSub-Question 2: If two players simultaneously catch and maintain control of the ball, and both players are inbounds, who is awarded possession?  A) The offensive player B) The defensive player C) The player who first touched the ball D) Possession is awarded to the team with the fewest points at the time\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}