{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"966e0ec5-4b76-5cae-960a-6a23d626f6f4","task_key":"bbeh~5fsportqa--full--966e0ec5-4b76-5cae-960a-6a23d626f6f4","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: How might the officiating crew determine that a double foul scenario occurred during a down where a defensive pass interference was also suspected?  A) By reviewing if both teams committed fouls independent of each other during the same down. B) By assessing if the defensive team committed pass interference after the offense had already committed a holding foul. C) By identifying if the offensive team committed a false start before the defensive player interfered with a receiver. D) By determining if the defensive pass interference occurred simultaneously with an offensive holding.\\nSub-Question 1: How can the officials confirm that the defensive team committed pass interference?  A) By checking if the defender made contact with the receiver before the ball was touched. B) By determining if the defender was playing the ball and not the receiver. C) By observing if the receiver was beyond the line of scrimmage when contact was made. D) By ensuring that the pass was catchable and the contact impeded the receiver's opportunity to catch the ball.\\nSub-Question 2: How would the officials ascertain that the offensive team committed a holding foul?  A) By establishing if an offensive player used his hands outside the defender's frame to restrict movement. B) By verifying if the offensive player's hands were inside the defender's shoulders while the defender was attempting to move past. C) By determining if the offensive player tackled the defender to the ground. D) By observing if the offensive lineman maintained a blocking position without grabbing the defender's jersey or pads.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}