{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"764e73d0-944c-5886-9bbb-4410dd3e04b5","task_key":"bbeh~5fsportqa--full--764e73d0-944c-5886-9bbb-4410dd3e04b5","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: What happens when a defensive player intercepts the ball in their own end zone and is subsequently tackled in the end zone, but a holding penalty is called on the offensive team during the interception?  A) The defensive team is awarded two points for a safety. B) The defensive team starts their possession from their own 20-yard line. C) The defensive team declines the penalty and takes possession at the spot of the tackle. D) The play is nullified, and the offensive team retains possession with a 10-yard penalty.\\nSub-Question 1: What are the outcomes if a player intercepts the ball in their own end zone and decides to take a knee?  A) The play results in a touchback, and the ball is placed on the 20-yard line. B) The intercepting team is awarded two points for a safety. C) The intercepting team starts their next drive from the spot of the kneel. D) The game clock stops until the next snap.\\nSub-Question 2: When is a team awarded a safety?  A) When an offensive player is tackled with the ball in their own end zone. B) When a defensive player intercepts the ball and is tackled in their own end zone without leaving it. C) When the ball goes out of bounds behind the offensive team's goal line due to their action. D) When an offensive penalty is enforced in their own end zone.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}