{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9cd91a96-22e2-5f7e-a791-2ad6e29830d5","task_key":"bbeh~5fsportqa--full--9cd91a96-22e2-5f7e-a791-2ad6e29830d5","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: During a critical NFL game, the quarterback of the home team throws a pass on a fourth-and-long situation late in the fourth quarter. The wide receiver catches the ball and is immediately tackled. One referee signals a first down, but another throws a flag for offensive pass interference. The instant replay shows that the receiver and the defender were in contact before the ball arrived. Which of the following outcomes is most consistent with the NFL rules regarding pass interference and spot of enforcement?  A) The penalty is accepted, the pass is deemed incomplete, and the offensive team is penalized with a loss of down. B) The penalty is declined, the pass is deemed complete, and the offensive team is awarded a first down. C) The penalty is accepted, the pass is deemed complete, but the offensive team is penalized 10 yards from the previous spot and repeats the down. D) The penalty is accepted, the pass is deemed complete, but the offensive team is penalized 10 yards from the spot of the foul and loses the down.\\nSub-Question 1: Which NFL rule defines the enforcement of penalties for offensive pass interference?  A) Rule 8, Section 5, Article 2 B) Rule 7, Section 5, Article 2 C) Rule 6, Section 3, Article 1 D) Rule 9, Section 4, Article 6\\nSub-Question 2: If a penalty for offensive pass interference is accepted, what is the consequence regarding the down?  A) The down is replayed with no changes. B) The down is considered complete, and the team moves to the next down. C) The down is replayed, and the offensive team is penalized with a loss of distance. D) The down is considered complete, and the offensive team loses the down.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}