{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"f5992add-96e8-5434-98f4-b2930b77accd","task_key":"bbeh~5fsportqa--full--f5992add-96e8-5434-98f4-b2930b77accd","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: During an NFL game, a wide receiver from Team A was running a route into the end zone when a defender from Team B made contact. The contact occurred simultaneously with a flag being thrown on the opposite side of the field. What infraction most likely caused the game officials to throw the flag, considering the position of the players and the timing of the events?  A) Offside by Team A's defensive player B) Pass interference by Team B's defender C) Illegal formation by Team A D) Holding by Team B's offensive lineman\\nSub-Question 1: If the contact between the wide receiver and the defender was deemed legal, which of the following is the most likely reason for the legality of the contact?  A) The contact occurred within five yards of the line of scrimmage B) The ball was already touched by another player C) The defender was playing the ball rather than the receiver D) The wide receiver initiated the contact\\nSub-Question 2: Assuming that the flag thrown on the opposite side of the field was for a separate infraction by Team B, which of the following infractions could it have been given the simultaneous timing of the events?  A) Encroachment by Team B's defensive lineman B) False start by Team B's wide receiver C) Delay of game due to Team B's quarterback D) Illegal use of hands by Team B's safety\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}