{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"bb8e965f-41a1-523c-a2d4-18ca7599debc","task_key":"bbeh~5fsportqa--full--bb8e965f-41a1-523c-a2d4-18ca7599debc","task_revision_id":"1","upstream_id":"","short_description":"You will be given a main question and two sub-questions. Each question comes…","config":"bbeh_sportqa","split":"full","body":"{\"input\":\"You will be given a main question and two sub-questions. Each question comes with multiple choices you can select from. For each question, select all the correct choices.\\n\\nMain Question: Why might a touchdown not be awarded even though a receiver catches the ball in the end zone during an American football game?  A) The receiver stepped out of bounds before making the catch. B) The ball was bobbled and not controlled before going out of bounds. C) The receiver was tackled in the end zone before completing the catch. D) The game clock expired before the ball was snapped.\\nSub-Question 1: What boundary rule specifics could negate a completed catch in the end zone?  A) The receiver's first contact with the ground was out of bounds after the catch. B) The receiver had both feet in bounds but was juggling the ball. C) The receiver maintained possession of the ball but landed on another player out of bounds. D) The receiver stepped on the end line before securing possession of the ball.\\nSub-Question 2: Under what circumstances does the game clock affect the scoring play in the end zone?  A) The game clock runs out while the ball is in the air heading towards the receiver. B) The play started after the game clock hit zero. C) The game clock expired during the play, but the snap was before the expiration. D) The game clock was not properly reset before the play started.\\n\\nFor each question, provide the answer as a concatenation of the correct choices.\\nSeparate the answers for the questions by comma.\\nFor example, if the correct choices for the main question are A and C, for the first sub-question is D and for the second sub-question are B and D, your final answer must be \\\"AC, D, BD\\\".\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}